Sequence-Level Divergence Optimization for Long-Context LLMs

TL;DR: Instead of measuring divergence at the token level, what if we estimate and constrain divergence over variable-length segments or entire sequences? This could better handle challenges of credit assignment and policy drift in long-context LLMs. An initial experiment might use DPPO with a sequence-level (or segment-level) KL or Total Variation constraint, compared against token-level baselines.

Research Question: Does enforcing trust regions on the sequence or segment level, as opposed to per-token, result in more stable and effective RL fine-tuning for tasks with long chains of reasoning?

Hypothesis: Segment/sequence-level divergence constraints will lead to more coherent and stable long-form outputs, as they respect the interdependencies among tokens and reduce the risk of local, myopic updates that can destabilize global behavior.

Experiment Plan: Extend DPPO to calculate and constrain divergence for contiguous token segments (e.g., sentences, reasoning chains) or full output sequences. Evaluate on benchmarks requiring multi-step reasoning (e.g., MATH500, GSM8K with long chain-of-thought). Compare learning dynamics, credit assignment accuracy, and final performance with token-level DPPO and PPO. Analyze whether segment-level constraints lead to fewer catastrophic shifts in reasoning style or factuality.

References:

• 1. Guo, Y., Xu, L., Liu, J., Ye, D., & Qiu, S. (2025). Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models. arXiv.org.