Unmasking the Hidden Subspaces: Visualizing and Interpreting Off-Principal Updates in RLVR

Research Question: How can we directly visualize and interpret the off-principal parameter updates induced by RLVR, and what does this reveal about the functional roles of these specialized subspaces?

Hypothesis: RLVR consistently updates a narrow, model-specific set of off-principal directions, and these subspaces correspond to functional modules (e.g., reasoning, memory) that are underutilized or left untouched by SFT.

Experiment Plan: - Implement tools to decompose parameter updates into principal and off-principal components (e.g., via SVD or eigendecomposition).

• Track and visualize the evolution of these components during RLVR training across different LLM architectures and tasks.
• Analyze whether the off-principal subspaces align with known functional circuits in language models (using probing or ablation).
• Compare with SFT and PEFT update patterns.
• Success would mean identifying stable, interpretable subspaces that RLVR exploits, potentially informing new forms of targeted regularization.

References: ['Zhu, H., Zhang, Z., Huang, H., Su, D., Liu, Z., Zhao, J., Fedorov, I., Pirsiavash, H., Sha, Z., Lee, J., Pan, D. Z., Wang, Z., Tian, Y., & Tai, K. S. (2025). The Path Not Taken: RLVR Provably Learns Off the Principals.', 'Cai, Y., Cao, D., Xu, X., Yao, Z., Huang, Y., Tan, Z., Zhang, B., Liu, G., & Fang, J. (2025). On Predictability of Reinforcement Learning Dynamics for Large Language Models. arXiv.org.']