Probing the Limits: Systematic Long-Horizon OPD Benchmark and Advanced Probing Tools

TL;DR: Let’s build a benchmark of ultra-long reasoning tasks and create new tools to probe where and why OPD breaks down. By systematically varying reasoning length and using diagnostic probes, we could map the true scalability frontier of OPD.

Research Question: Where does on-policy distillation begin to fail as reasoning horizon increases, and what token-level or trajectory-level phenomena signal impending collapse?

Hypothesis: There exists a critical reasoning horizon beyond which OPD’s progressive alignment mechanism collapses, characterized by divergence in token alignment, increased gradient instability, or probability mass fragmentation—detectable by fine-grained diagnostic probes.

Experiment Plan: Construct synthetic and real reasoning datasets with systematically increasing chain-of-thought lengths (e.g., multi-step math puzzles, algorithmic reasoning). Develop new probing tools: e.g., visualize token alignment, measure per-token KL, track entropy collapse, and analyze gradient variance as training progresses. Run OPD training across horizons, documenting when and how failure modes (as described by Li et al., 2026 and Luo et al., 2026) emerge. Use advanced analysis (e.g., Markov models, sequence clustering) to pinpoint specific failure transitions. Release benchmark and diagnostic suite for community use.

References:

• Li, Yaxuan, et al. (2026). Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe.
• Luo, Feng, Chuang, Yu-Neng, Wang, Guanchu, et al. (2026). Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models.