Adaptive Headroom Estimation: Dynamic Curriculum Scheduling for Reasoning Gains in RL

TL;DR: What if we could automatically figure out how much "room to grow" (headroom) a model has at any point and adapt our RL curriculum on the fly? The initial experiment would develop an algorithm that periodically estimates reasoning headroom and dynamically adjusts RL task difficulty accordingly, hypothesizing this yields more robust and efficient reasoning improvements.

Research Question: Can real-time estimation of a model’s reasoning headroom be used to dynamically adapt RL curricula, thereby maximizing capability gains and minimizing wasted training effort?

Hypothesis: If the RL curriculum continuously targets a model's edge of competence—determined via dynamic headroom estimation—models will achieve higher reasoning improvement per unit of compute compared to static or hand-crafted curricula.

Experiment Plan: - Develop a metric or probe that quantifies "headroom" (the gap between current ability and maximum achievable performance on reasoning tasks).

• Train multiple models using synthetic reasoning tasks as in Zhang et al. (2025), but with an adaptive curriculum that selects tasks at the model’s current boundary of competence.
• Compare against static curriculum and fixed-difficulty RL training.
• Measure overall reasoning improvement (pass@128), learning efficiency (performance per step), and generalization to harder or novel compositions.

References:

• Zhang, C., Neubig, G., & Yue, X. (2025). On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models.