OPSDC Meets the Latent Space: Self-Distillation for Implicit Continuous Reasoning Compression

TL;DR: Let’s see if OPSDC’s magic works not just on language, but also in the hidden spaces where the model “thinks” outside of words! Train models to compress and distill concise reasoning in the latent (continuous) space, and compare its efficiency and accuracy to language-based approaches like CODI.

Research Question: Can OPSDC-style self-distillation be applied in the latent representation space, achieving compression and accuracy gains for implicit (non-linguistic) reasoning models?

Hypothesis: Aligning concise latent reasoning traces through self-distillation (as in CODI) will further reduce redundancy and improve generalization, potentially enabling faster and more robust implicit reasoning.

Experiment Plan: Extend OPSDC to continuous reasoning models by using hidden state alignment instead of token-level KL (following CODI [CODI, 3]). Jointly train explicit (natural language) and implicit (latent) reasoning branches, with self-distillation enforcing conciseness in both. Benchmark on GSM8K, MATH500, and synthetic datasets, comparing accuracy, compression rate, and robustness to noise with CODI and explicit OPSDC. Analyze interpretability and error patterns in compressed latent reasoning traces.

References:

• 1. Sang, H., Xu, Y., Zhou, Z., He, R., Wang, Z., & Sun, J. (2026). On-Policy Self-Distillation for Reasoning Compression.
• 2. Shen, Z., Yan, H., Zhang, L., Hu, Z., Du, Y., & He, Y. (2025). CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation. Conference on Empirical Methods in Natural Language Processing.