Curriculum-Driven OPD: Dynamic Task Sequencing to Unlock Long-Horizon Reasoning

TL;DR: If OPD struggles with long, hard problems, could we train models on a carefully sequenced curriculum that gradually increases reasoning horizon and complexity? Try starting with short reasoning chains, then mixing in longer problems as the student aligns, to see if this helps OPD scale.

Research Question: Does a dynamically adaptive curriculum—progressively increasing problem horizon and complexity—enable on-policy distillation to succeed on tasks that otherwise exceed the current scalability limits of OPD?

Hypothesis: A curriculum learning framework, where the student is distill-trained first on short or easy reasoning paths and then on incrementally longer and harder ones, will facilitate stable OPD training and enable scaling to longer-horizon reasoning tasks.

Experiment Plan: Construct a suite of reasoning datasets ordered by chain-of-thought length and difficulty (e.g., math word problems and multi-hop QA). Implement a curriculum scheduler that selects training samples based on student proficiency (e.g., using reward signals or error rates). Track OPD training dynamics and performance as the curriculum advances, both with and without curriculum (baseline: random sampling). Evaluate whether curriculum-OPD achieves higher accuracy or stability on long-horizon benchmarks, and analyze token-alignment trajectories. Optionally, compare to techniques from Video-OPD (Li et al., 2026) that use teacher-validated disagreement focusing as a related form of curriculum.

References:

• Li, Yaxuan, et al. (2026). Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe.
• Li, Jiaze, Yin, Hao, Xu, Haoran, et al. (2026). Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation. arXiv.org.