Cross-Domain Self-Distillation: Transferring Precision-Exploration Insights to Natural Language Understanding

TL;DR: What if the SSD trick also improved LLMs on tasks like reading comprehension or sentiment analysis, not just code? We'll try SSD with context-dependent sampling on NLU benchmarks and see if similar gains (especially on hard/outlier samples) appear.

Research Question: Can SSD, as characterized by precision-exploration conflict resolution, improve performance on non-code LLM tasks such as natural language understanding (NLU), and do the gains similarly concentrate on harder cases?

Hypothesis: The mechanisms observed in code generation extend to NLU; SSD will yield greater improvements on complex or ambiguous NLU tasks, with token distribution shifts reflecting task-specific needs for precision or exploration.

Experiment Plan: Select several NLU benchmarks (e.g., SQuAD, GLUE, sentiment classification). Apply SSD: sample model outputs under varying temperatures/truncations, then fine-tune. Stratify tasks by "difficulty" (e.g., ambiguous questions, rare labels). Compare performance to vanilla fine-tuning and analyze gain distribution. Examine token probability changes and error types pre/post-SSD.

References:

• Zhang, R., Bai, R., Zheng, H., Jaitly, N., Collobert, R., & Zhang, Y. (2026). Embarrassingly Simple Self-Distillation Improves Code Generation.