Adaptive Vocabulary-Aware Divergence Constraints for LLM RL

TL;DR: What if the divergence constraint itself was dynamically tailored to each token’s frequency or semantic role? Imagine an RL algorithm that tightens or loosens trust regions based on whether a token is rare or common, potentially improving both sample efficiency and stability. A first experiment could involve modifying DPPO so that, for each training batch, the divergence constraint is adjusted on a per-token basis based on token frequency statistics or entropy.

Research Question: Can adaptively scaling the divergence constraint in DPPO according to token frequency or contextual importance further improve training stability and generalization for LLMs?

Hypothesis: By relaxing constraints for low-frequency (rare) tokens—thus allowing more exploration—and tightening them for high-frequency (common or function) tokens, the RL agent can avoid over-penalizing rare but important behaviors while maintaining global policy stability.

Experiment Plan: Implement a variant of DPPO where the divergence threshold is modulated by token frequency (e.g., from the training corpus) or dynamic per-batch statistics. Compare against fixed-constraint DPPO and PPO on reasoning and dialogue LLM benchmarks (e.g., GSM8K, MATH500). Key metrics: training stability (variance in rewards/loss), final accuracy, and diversity of rare-token generations. Analyze whether this approach better preserves or improves rare token utility without sacrificing overall model coherence.

References:

• 1. Park, J. R., Kim, J., Kim, G., Jo, J., Choi, S., Cho, J., & Ryu, E. K. (2025). Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models. arXiv.org.
• 2. Becker, P., Freymuth, N., Thilges, S., Otto, F., & Neumann, G. (2025). TROLL: Trust Regions improve Reinforcement Learning for Large Language Models. arXiv.org.