A Peer Review Feedback Loop: Integrating AI Error Detection into Bi-Directional Quality Assurance Frameworks

TL;DR: Imagine a peer review system where AI not only checks for mistakes but also helps reviewers and authors learn from them—making the whole system smarter! The study would trial AI-powered error reporting and correction within a two-way review-feedback system.

Research Question: Can integrating LLM-based error detection into a bi-directional peer review process improve review quality, reduce error prevalence, and enhance transparency in AI research publishing?

Hypothesis: A peer review platform that includes automated, transparent AI error reports alongside human reviews—plus structured feedback from authors—will improve error correction rates and overall trust in the publication process.

Experiment Plan: - Setup: Modify an open-source conference management system (e.g., OpenReview) to integrate LLM error checkers and structured author/reviewer feedback.

• Participants: Run a pilot during a special track or workshop at a major AI conference.
• Metrics: Track error detection/correction rates, reviewer/author satisfaction, and downstream citation impact.
• Expected Outcomes: Demonstrated improvements in error correction, accountability, and community trust, with lessons for scaling to mainstream conferences.

References:

• Kim, J., Lee, Y., & Lee, S. (2025). Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards. arXiv.org.
• Yu, S., Luo, M., Madusu, A., Lal, V., & Howard, P. (2025). Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Review.