Reasoning-Chain-Aware Self-Distillation for Complex Code Generation

TL;DR: What if we used the model’s own reasoning chains as both source and target for SSD, not just the final code? We'll generate and fine-tune on intermediate reasoning steps, testing if this enhances solution completeness and robustness on multi-step problems.

Research Question: Does incorporating self-distillation of intermediate reasoning traces (not just final outputs) enhance code LLM performance, especially on multi-step and hard problems?

Hypothesis: By distilling both reasoning chains and answers, the model will internalize better intermediate representations, reducing common failure modes like incomplete solutions noted by Xue et al. (2025).

Experiment Plan: Use a "thinking" LLM to generate reasoning traces alongside code outputs. Apply SSD to both: fine-tune the model with its own sampled traces and solutions. Evaluate on benchmarks that require step-by-step reasoning (e.g., BigCodeBench). Measure completeness, logical correctness, and robustness (via human and automated metrics). Compare to SSD on final solutions only.

References:

• Xue, H., Uddin, G., & Wang, S. (2025). An Empirical Study of Reasoning Steps in Thinking Code LLMs. arXiv.org.
• Zhang, R., Bai, R., Zheng, H., Jaitly, N., Collobert, R., & Zhang, Y. (2026). Embarrassingly Simple Self-Distillation Improves Code Generation.