Benchmarking Reasoning Judge Vulnerability with Real-World Adversarial Datasets

TL;DR: Let's build and use new datasets full of tricky, real-world examples specifically designed to expose the weaknesses of reasoning judges and judge-trained policies. If models trained with reasoning judges can still be fooled, we’ll know where to improve next.

Research Question: How do reasoning LLM-judges and judge-aligned policies perform on adversarial datasets derived from real-world, non-verifiable tasks, compared to synthetic benchmarks?

Hypothesis: Current reasoning LLM-judges and their aligned policies will be more susceptible to adversarial exploitation in diverse, realistic datasets than on synthetic, controlled tasks, revealing important gaps in current alignment methods.

Experiment Plan: Construct or curate new adversarial datasets in domains like social dialogue, creative writing, or legal reasoning, where correctness is inherently non-verifiable. Design adversarial examples specifically to exploit reasoning patterns or overfit judge scoring rubrics. Benchmark LLM-judge-aligned policies and baseline models on these datasets, measuring adversarial success rates, alignment, and robustness. Use findings to propose next-generation judge architectures or training regimes.

References:

• Liu, Y., Yu, Y., Su, D., Wang, S., Wang, X., Jiang, S., Liu, B., Cohan, A., Tian, Y., & Chen, Z. (2026). Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training.
• Huertas-García, Á., Martín, A., Huertas-Tato, J., & Camacho, D. (2024). Camouflage is all you need: Evaluating and Enhancing Language Model Robustness Against Camouflage Adversarial Attacks. arXiv.org.
• Khosrojerdi, A., Bigdeli, A., Hamidi Rad, R., Zihayat, M., Clarke, C., & Bagheri, E. (2025). Datasets for Supervised Adversarial Attacks on Neural Rankers. International Conference on Information and Knowledge Management.