Self-Distillation Outlier Tracking: Diagnosing and Amplifying Gains on Hard Code Generation Cases

TL;DR: What if we could automatically spot which code problems benefit most from SSD, and then tailor the distillation process to maximize those gains? Let's develop a dynamic SSD pipeline that detects hard/outlier problems and adaptively tweaks sampling and training schedules, then measure if such targeting further closes the gap on "difficult" code challenges.

Research Question: Can dynamic detection and targeted self-distillation on hard or outlier code generation problems lead to greater improvements than applying SSD uniformly across all samples?

Hypothesis: By identifying and focusing SSD on hard or outlier problems—perhaps using difficulty heuristics, error clustering, or loss-based anomaly detection—the model will achieve higher pass rates, especially on the long tail of challenging problems, compared to uniform SSD.

Experiment Plan: Use LiveCodeBench and similar code-generation benchmarks; label problems by difficulty (e.g., pass rates, error types, reasoning step analysis). Run standard SSD as a baseline. Develop an "outlier detector" (e.g., using failure patterns, high loss, or unique reasoning traces as per Xue et al., 2025). Implement an adaptive SSD scheme: for detected hard/outlier problems, increase sampling diversity or fine-tuning weight. Evaluate pre- and post-intervention pass@1, especially on hard cases. Analyze if gains concentrate even more on the hardest tasks. Perform ablation to test which detection/adaptation strategies are most effective.

References:

• Zhang, R., Bai, R., Zheng, H., Jaitly, N., Collobert, R., & Zhang, Y. (2026). Embarrassingly Simple Self-Distillation Improves Code Generation.
• Xue, H., Uddin, G., & Wang, S. (2025). An Empirical Study of Reasoning Steps in Thinking Code LLMs. arXiv.org.