Counterfactual Self-Distillation: Learning from Alternate Realities in RL with Rich Feedback

TL;DR: What if the model could learn not just from its actual feedback, but also from "what might have been"? We train an agent to imagine alternative outcomes for its actions and distill lessons from both real and counterfactual feedback, hypothesizing that this broadens generalization and improves robustness. An initial experiment could generate, for each failed attempt, plausible alternative feedback sequences using the model itself or a secondary LLM, and distill both actual and counterfactual next-token predictions into the policy.

Research Question: Can augmenting SDPO with synthetic, counterfactual feedback derived from alternative action sequences further improve policy robustness and generalization in RL with rich feedback?

Hypothesis: By exposing the agent to counterfactual feedback—i.e., what would have happened if it had taken different actions—the model can better understand the causal structure of the environment, leading to improved sample efficiency and performance, particularly in out-of-distribution or novel tasks.

Experiment Plan: Extend the SDPO framework to generate, for each rollout, alternative plausible sequences (counterfactuals) using beam search or a secondary generative model. Produce synthetic feedback for these counterfactuals (e.g., "If you had tried X instead of Y, you would have received error Z"). Distill next-token predictions from both the real and counterfactual feedback-conditioned models into the main policy. Evaluate on code and math reasoning tasks, comparing standard SDPO, counterfactual-augmented SDPO, and RLVR baselines. Key measurements: accuracy, generalization to unseen tasks, and sample efficiency, with special attention to performance on novel or adversarial examples.

References:

• Hubotter, J., Lubeck, F., Behric, L., Baumann, A., Bagatella, M., Marta, D., Hakimi, I., Shenfeld, I., Kleine Buening, T., Guestrin, C., & Krause, A. (2026). Reinforcement Learning via Self-Distillation.
• Li, J., Shi, H., Wu, H., Zhao, C., & Hwang, K.-S. (2024). Eliminating Primacy Bias in Online Reinforcement Learning by Self-Distillation. IEEE Transactions on Neural Networks and Learning Systems.