Process-Reward Interpretability: Unpacking the Impact of Step-Level Rewards on Reasoning Trace Quality

TL;DR: Can we make RL's influence on reasoning steps more transparent and interpretable? The experiment would combine process-level rewards with automated reasoning trace evaluation (using tools like AutoRace), aiming to dissect how rewards at each step shape reasoning fidelity and generalization.

Research Question: How do process-level (step-by-step) rewards influence the quality, diversity, and generalization of reasoning traces in LMs, and can automated trace evaluation provide actionable feedback for reward design?

Hypothesis: Process-level rewards not only improve final answer accuracy but also lead to more interpretable, diverse, and robust reasoning traces, as measured by automated, task-specific reasoning trace metrics.

Experiment Plan: - Train models with and without process-level rewards on synthetic reasoning tasks.

• Use an automated reasoning trace evaluation tool (e.g., AutoRace from Hao et al., 2024) to assess trace quality, diversity, and error localization.
• Compare not just final answer accuracy but also trace interpretability, error types, and generalization to new reasoning tasks.

References:

• Zhang, C., Neubig, G., & Yue, X. (2025). On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models.
• Hao, S., Gu, Y., Luo, H., et al. (2024). LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models. arXiv.org.