Factuality-Aware Reward Modeling for Stepwise Mathematical Object Construction

TL;DR: What if our models could check each step of a math solution for correctness, not just the final answer? This experiment would train LLMs using stepwise factuality checks as part of their reward signal, targeting the reduction of “hallucinated” intermediate steps in formal math derivations.

Research Question: Can incorporating explicit factuality verification at each reasoning step during on-policy training and aggregation reduce logical errors and hallucinations in mathematical object derivation compared to standard reward modeling approaches?

Hypothesis: Stepwise factuality-aware training will decrease the frequency and propagation of intermediate reasoning errors, resulting in more reliable and interpretable solutions on Principia and related benchmarks.

Experiment Plan: - Integrate automated factuality checks (possibly using external verifiers or code execution) into the reward modeling for each intermediate step.

• Train models on Principia using this enhanced reward signal.
• Compare with baseline models in terms of error propagation, hallucination rates, and end-to-end accuracy.
• Evaluate human interpretability and trust in the generated solution paths.

References:

• Li, J., & Ng, H. T. (2025). Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models.
• Aggarwal, P., Ghazvininejad, M., Kim, S., Kulikov, I., Lanchantin, J., Li, X., Li, T., Liu, B., Neubig, G., Ovalle, A., Saha, S., Sukhbaatar, S., Welleck, S., Weston, J., Whitehouse, C., Williams, A., Xu, J., Yu, P., Yuan, W., Zhang, J., & Zhao, W. (2026). Reasoning over mathematical objects: on-policy reward modeling and test time aggregation.