Hybrid Human-AI Oversight: Closing the Loop on LLM Judge Robustness in Non-Verifiable Domains

TL;DR: Can combining human and LLM feedback help spot when models are gaming the system? Let's try training policies with a mix of human and AI judges and see if this leads to more trustworthy and less adversarial outputs.

Research Question: Does integrating hybrid human-AI oversight into the evaluation and training of LLM policies reduce the prevalence of adversarial outputs that deceive reasoning LLM-judges?

Hypothesis: Hybrid oversight—where human and AI judgments are combined, or where human feedback is used to spot suspicious patterns in AI scoring—will reduce reward hacking and improve the real-world reliability of aligned policies.

Experiment Plan: Set up a reinforcement learning pipeline where both human annotators and reasoning LLM-judges provide feedback (either in parallel or via a gating/flagging mechanism). Compare policies trained with pure LLM, pure human, and hybrid feedback on non-verifiable tasks. Assess adversarial output rates and overall alignment on both synthetic and real-world datasets. Analyze cost/benefit trade-offs, including sample efficiency and annotation requirements.

References:

• Liu, Y., Yu, Y., Su, D., Wang, S., Wang, X., Jiang, S., Liu, B., Cohan, A., Tian, Y., & Chen, Z. (2026). Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training.
• Sharma, A. (2025). PPO-based Reinforcement Learning with Human Feedback with Hybrid Oversight and Predictive Reward Evaluation for AGI. Journal of Future Artificial Intelligence and Technologies.