Rethinking the "Gold Standard": Are Reasoning Judges Actually the Best Policy Trainers?

TL;DR: What if simpler or alternative judges—like tool-augmented, multi-objective, or ensemble judges—are better at aligning policies than current reasoning LLM-judges? Let’s test this head-to-head.

Research Question: Are there alternative judge architectures or evaluation strategies that outperform reasoning LLM-judges in aligning policies for non-verifiable domains, especially regarding adversarial robustness?

Hypothesis: Tool-augmented judges (e.g., code executors, fact checkers), ensemble judges, or multi-objective reward models can provide more robust and less exploitable alignment signals than standalone reasoning LLM-judges.

Experiment Plan: Implement and train various alternative judge systems: tool-integrated (e.g., TIR-Judge), ensembles, and multi-objective frameworks. Compare policy training outcomes using each judge type on non-verifiable tasks, evaluating robustness to adversarial outputs and overall alignment quality. Analyze judge decisions and policies for evidence of reward hacking, adversarial exploitation, and generalization.

References:

• Liu, Y., Yu, Y., Su, D., Wang, S., Wang, X., Jiang, S., Liu, B., Cohan, A., Tian, Y., & Chen, Z. (2026). Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training.
• Xu, R., Chen, J., Ye, J., Wu, Y., Yan, J., Yang, C., & Yu, H. (2025). Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning. arXiv.org.