Self-Distillation with Feedback Compression: Learning from Summarized Mistakes

TL;DR: Could the model benefit from feedback that's been automatically summarized or abstracted, rather than raw verbose traces? We propose a variant of SDPO where the model first compresses rich feedback into high-level summaries or error categories before distillation, hypothesizing that this will improve learning efficiency and generalization. As a first step, compare distillation from raw versus summarized feedback on code and math tasks.

Research Question: Does summarizing or abstracting rich textual feedback into concise error categories or lessons improve the efficiency and effectiveness of self-distillation in RL?

Hypothesis: Condensing feedback into high-level summaries or error taxonomies removes noise and highlights core learning signals, enabling the model to focus on generalizable lessons rather than task-specific details, thus accelerating learning and improving transferability.

Experiment Plan: Train or fine-tune a feedback summarizer (either a rule-based system or a small LLM) to convert detailed feedback into concise summaries or error types. Use SDPO to distill next-token predictions conditioned on both raw and summarized feedback. Compare sample efficiency, final accuracy, and policy generalization to new error types or feedback formats. Optionally, explore curriculum learning by progressively increasing the abstraction level of feedback over training.

References:

• Hubotter, J., et al. (2026). Reinforcement Learning via Self-Distillation.
• Li, A., Wang, Y., Yuan, Z., Jegelka, S., & Wang, Y. (2025). LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs. arXiv.org.
• Chen, L., Shen, B., & Hong, J. (2025). A multi-task deep reinforcement learning framework based on curriculum learning and policy distillation for quadruped robot motor skill training. Systems Science & Control Engineering.