Reviewer in the Loop: Human-AI Hybrid Workflows to Detect and Mitigate Indirect Prompt Injection

TL;DR: What if we combine the strengths of humans and LLMs? Let’s experiment with workflows where suspicious LLM reviews are flagged for human oversight, or where humans get explainability cues about potential manipulations.

Research Question: Can hybrid workflows that incorporate human oversight and LLM explainability reduce the impact of indirect prompt injection on peer review outcomes?

Hypothesis: Integrating human reviewers in the loop, supplemented by automated detection and explainability tools, can significantly decrease successful adversarial manipulation rates compared to fully automated LLM review.

Experiment Plan: Build a system that flags LLM-generated reviews suspected of being manipulated (using metrics like sudden score changes or text similarity to known attack patterns). In a user study, give human reviewers access to flagged reviews and explainability visualizations. Compare decision outcomes and error rates between (a) LLM-only, (b) human-only, and (c) hybrid workflows. Collect qualitative feedback on trust and usability.

References:

• Sahoo, D., Prasad, M., Majhi, V., Singh, J., Chamola, V., Sinha, Y., Mandal, M., & Kumar, D. (2025). When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injection.
• Li, J., Li, Y., Hu, X., Gao, M., & Wan, X. (2025). Aspect-Guided Multi-Level Perturbation Analysis of Large Language Models in Automated Peer Review. arXiv.org.
• Hosseini, M., & Horbach, S. (2023). Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review. Research Integrity and Peer Review.