Reverse-Engineered Pre-Training: Synthetic Disentanglement for Causal Attribution in RL Gains

TL;DR: What if we could systematically "erase" specific reasoning skills from pre-training, then see exactly how much RL (or mid-training) can recover them? The experiment would create synthetic tasks with selectively omitted reasoning operations during pre-training, then test RL’s power to recover or build those skills.

Research Question: To what extent can RL (or mid-training) recover or learn reasoning skills that were deliberately omitted during pre-training, and does this differ from simply amplifying latent capabilities?

Hypothesis: RL can only partially compensate for omitted reasoning skills; certain atomic operations must be present in pre-training for RL to build upon, suggesting a lower bound of necessary pre-training exposure for RL effectiveness.

Experiment Plan: - Design synthetic reasoning datasets where certain operations (e.g., addition, subtraction, multi-step compositions) are omitted from pre-training data.

• Conduct mid-training and RL on these models, measuring ability to acquire the omitted skills.
• Analyze whether RL recovers the missing skills, how quickly, and with what data efficiency.
• Compare to models with full pre-training exposure.

References:

• Zhang, C., Neubig, G., & Yue, X. (2025). On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models.