Beyond the Formula: Detecting Subjective and Contextual Errors in AI Publications Using Multimodal Large Language Models

TL;DR: What if AI could spot not just factual goofs, but also subtle, context-dependent errors or misleading claims in research papers? This study would train a new LLM system to flag subjective mistakes and misinterpretations that require domain context or scientific judgment.

Research Question: Can multimodal LLMs, enhanced with domain-specific retrieval and contextual reasoning, reliably detect subjective or context-sensitive errors in AI research papers?

Hypothesis: Multimodal LLMs, especially when augmented with retrieval from domain-specific scientific databases and prior literature, will outperform current models in identifying nuanced, context-dependent mistakes that go beyond objective errors.

Experiment Plan: - Data: Curate a benchmark of AI papers with annotated subjective/contextual errors (e.g., misinterpretation of related work, overstated claims).

• Methodology:
  • Develop a RAG-augmented LLM pipeline (building on ACURAI/RAG approaches) tailored for scientific critique.
  • Compare detection performance on subjective errors versus baseline LLMs and human reviewers.
• Expected Outcomes: Demonstrated improvement in detecting errors that require understanding scientific context, citation relevance, or experimental design critiques.

References:

• Anghelescu, A., Munteanu, C., Anghelescu, L. A. M., & Onose, G. (2025). “A Midsummer Night’s Dream” quest for truth: From ChatGPT “hallucinations” to RAG reasoning and ACURAI precision — a scoping review on detection, minimizing, and (almost) complete error elimination and enhancing Large Language Models' reliability. Balneo and PRM Research Journal.
• Cheng, P. J., Hu, F. Y., Chen, L. Y., Liu, J. Y., Wu, J. H., & Chen, W. L. (2025). Generative artificial intelligence in ophthalmology research writing: A comprehensive review of applications, detection tools, and ethical considerations. Taiwan Journal of Ophthalmology.