Human vs. AI: A Controlled Trial of Error Detection and Correction in Scientific Peer Review

TL;DR: Let’s run a bake-off to see whether humans or AI (or both together) are best at catching and fixing mistakes in research papers! The experiment would compare the effectiveness of expert human reviewers, LLMs, and hybrid teams in error detection and correction.

Research Question: How do the accuracy, speed, and types of errors detected differ between expert human reviewers, advanced LLM-based checkers, and human-AI collaborative teams when reviewing scientific AI papers?

Hypothesis: Hybrid teams (AI + human) will outperform either group alone, with AI excelling in objective error detection and humans providing critical context and judgment for ambiguous cases.

Experiment Plan: - Setup: Select a representative sample of AI papers with known, seeded errors (both objective and subjective).

• Groups: Assign error-checking tasks to (a) human experts, (b) LLMs, and (c) human-AI teams with structured collaboration.
• Metrics: Measure precision, recall, time-to-detection, and error category coverage.
• Expected Outcomes: Hybrid teams will show superior performance, especially in complex or nuanced error categories, and will demonstrate improved efficiency.

References:

• Schmidt, R. A., Seah, J., Cao, K., Lim, L. J., Lim, W., & Yeung, J. M. (2024). Generating Large Language Models for Detection of Speech Recognition Errors in Radiology Reports. Radiology: Artificial Intelligence.
• Seo, H., Lee, S., Lee, D., & Xiong, A. (2024). Reliability Matters: Exploring the Effect of AI Explanations on Misinformation Detection with a Warning. International Conference on Web and Social Media.
• Awasthi, A., Le, N., Deng, Z., Wu, C. C., & Nguyen, H. (2025). Collaborative Integration of AI and Human Expertise to Improve Detection of Chest Radiograph Abnormalities. Radiology: Artificial Intelligence.