Hybrid Human-AI Missing Information Detection: Crowdsourcing for Negative Space Calibration

TL;DR: What if we combine LLMs with humans—using hybrid systems where humans "teach" the model how to notice what's missing, especially in ambiguous cases? An initial study could use crowdsourced annotations to improve LLM calibration on absence detection, as in misinformation detection.

Research Question: Can hybrid human-AI approaches, leveraging crowdsourced judgments, significantly enhance LLMs' ability to detect missing information, especially in complex or ambiguous contexts?

Hypothesis: Incorporating human-labeled data on missing content—paired with model predictions—enables more accurate calibration of LLMs, reducing false negatives and improving overall absence detection performance.

Experiment Plan: - Data Collection: Use crowdsourcing to annotate missing information in AbsenceBench and additional real-world datasets (e.g., news, clinical reports).

• Hybrid System: Implement ensemble methods (e.g., Model First, Worker First, Meta Vote) combining LLM and human assessments.
• Evaluation: Measure improvement over pure LLM and pure human baselines.
• Analysis: Examine cases where hybrid methods outperform both humans and models.
• Expected Outcome: Hybrid strategies will show superior accuracy and robustness, especially for subtle or context-dependent omissions.

References:

• Zeng, X., La Barbera, D., Roitero, K., Zubiaga, A., & Mizzaro, S. (2024). Combining Large Language Models and Crowdsourcing for Hybrid Human-AI Misinformation Detection. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
• Mousavian, M., Abbasiantaeb, Z., Aliannejadi, M., & Crestani, F. (2025). Towards Fair Rankings: Leveraging LLMs for Gender Bias Detection and Measurement. International Conference on the Theory of Information Retrieval.