Human-in-the-Loop Hypothesis Evolution: Crowdsourcing Unexpected LLM Behaviors

Building on the democratizing ethos of Evalverse (Kim et al., 2024) and the hypothesis-generation focus of Koneru et al. (2023), this proposal leverages human feedback as a primary driver of hypothesis evolution. Instead of relying solely on expert-designed benchmarks, the platform would allow users (e.g., clinicians using PEACH or researchers probing LLMs in scientific discovery) to submit outputs that deviate from their expectations. The system then prompts users to articulate plausible hypotheses about underlying causes (e.g., “the model may be overfitting to recent literature” or “the model misinterprets ambiguous terminology”). These hypotheses are then formalized, tested, and, if validated, incorporated into future evaluation cycles. This “citizen science” approach could radically expand the diversity of evaluation scenarios and ensure that LLM assessment keeps pace with real-world deployment challenges.

References:

1. Evalverse: Unified and Accessible Library for Large Language Model Evaluation. Jihoo Kim, Wonho Song, Dahyun Kim, Yunsu Kim, Yungi Kim, Chanjun Park (2024). Conference on Empirical Methods in Natural Language Processing.
2. Real-world Deployment and Evaluation of PErioperative AI CHatbot (PEACH) - a Large Language Model Chatbot for Perioperative Medicine. Yuhe Ke, Liyuan Jin, Kabilan Elangovan, Bryan Wen Xi Ong, Chin Yang Oh, Jacqueline Sim, Kenny Wei-Tsen Loh, Chai Rick Soh, Jonathan Ming Hua Cheng, Aaron Kwang Yang Lee, D. Ting, Nan Liu, H. Abdullah (2024). arXiv.org.
3. Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences. S. Koneru, Jian Wu, Sarah M. Rajtmajer (2023). International Conference on Language Resources and Evaluation.