Efficient Low-Rank Divergence Estimation for LLM RL

TL;DR: Can we use low-rank approximations or decompositions to estimate policy divergence more efficiently, especially for massive vocabularies? Imagine leveraging matrix factorization or sketching to compute approximate KL or TV divergence with minimal memory. A concrete experiment could compare low-rank DPPO variants against top-K and binary approximations in terms of speed and alignment quality.

Research Question: Can low-rank matrix approximations of policy distributions yield accurate enough divergence estimates for DPPO, enabling further scaling of RL fine-tuning with negligible overhead?

Hypothesis: Low-rank projections of the policy's output logits or probabilities can provide sufficiently accurate proxies for divergence calculation, further reducing computational cost and enabling RL on even larger models or longer output sequences.

Experiment Plan: Develop and implement a low-rank DPPO variant using SVD, PCA, or randomized sketching to approximate the policy distributions before divergence computation. Run experiments on medium- and large-scale LLMs, comparing training time, GPU memory usage, and RL outcomes to top-K/binary DPPO and full-divergence baselines. Key analyses: quality of divergence estimates, stability of policy updates, and empirical sample efficiency.

References:

• 1. (Current Paper) Rethinking the Trust Region in LLM Reinforcement Learning.
• 2. Huang, L., Li, S., Anupam, S., Lee, I., & Bastani, O. (2025). Effective Reinforcement Learning for Reasoning in Language Models. arXiv.org.