Reasoning Trace Length as a Proxy for Robustness: A Systematic Investigation

TL;DR: Does a longer or more detailed reasoning chain always mean better generalization? Systematically vary and constrain reasoning trace lengths in LLMs to see how this influences OOD performance and uncertainty expression, challenging the assumption that shorter is always better.

Research Question: What is the relationship between reasoning trace length, uncertainty expression, and OOD robustness in LLMs, and can optimal trace lengths be learned or adapted dynamically to maximize generalization?

Hypothesis: There exists a non-monotonic relationship between reasoning trace length and OOD robustness: both overly short and overly verbose traces can harm performance, and the optimal trace length may depend on task distribution.

Experiment Plan: - Setup: Train LLMs with explicit constraints or incentives on reasoning trace length (e.g., reward for brevity, verbosity, or adaptive length based on uncertainty).

• Data: Apply to math reasoning datasets with clear in-domain and OOD splits.
• Measurements: Analyze performance, trace length, and uncertainty markers, searching for trade-offs.
• Expected Outcomes: Adaptive trace length controls—potentially informed by uncertainty—will outperform fixed-length approaches, offering a new perspective on reasoning optimization beyond what Kim et al. (2026) observed.

References:

• Kim, J., Luo, X., Kim, M., Lee, S., Kim, D., Jeon, J., Li, D., & Yang, Y. (2026). Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?