Dynamic Hypothesis Tracking for Real-Time LLM Evaluation

Most current hypothesis-guided evaluation frameworks for LLMs rely on static sets of evaluation questions or benchmarks (see, e.g., Kumbhar et al., 2025; Song et al., 2025). However, real-world deployment—such as in the PEACH perioperative chatbot (Ke et al., 2024)—shows that LLM behaviors and potential failure modes shift over time as usage contexts evolve. This research idea proposes a system that continuously tracks deviations from expected behavior (cf. Zhu et al., 2024) and iteratively refines evaluation hypotheses in response. For example, if an LLM begins to exhibit new types of hallucinations in a clinical setting, the system would generate new hypotheses about root causes and update evaluation protocols on-the-fly. This is distinct from prior work by making hypothesis formation itself an ongoing, data-driven process, rather than a one-off design decision. The potential impact is a much more agile and resilient evaluation pipeline—critical for safe, long-term deployment of LLMs in sensitive or rapidly changing domains.

References:

1. Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation. Qin Zhu, Qingyuan Cheng, Runyu Peng, Xiaonan Li, Tengxiao Liu, Runyu Peng, Xipeng Qiu, Xuanjing Huang (2024). Conference on Empirical Methods in Natural Language Processing.
2. Real-world Deployment and Evaluation of PErioperative AI CHatbot (PEACH) - a Large Language Model Chatbot for Perioperative Medicine. Yuhe Ke, Liyuan Jin, Kabilan Elangovan, Bryan Wen Xi Ong, Chin Yang Oh, Jacqueline Sim, Kenny Wei-Tsen Loh, Chai Rick Soh, Jonathan Ming Hua Cheng, Aaron Kwang Yang Lee, D. Ting, Nan Liu, H. Abdullah (2024). arXiv.org.
3. LLM Agent Swarm for Hypothesis-Driven Drug Discovery. Kevin Song, Andrew Trotter, Jake Y. Chen (2025). arXiv.org.
4. Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents. Shrinidhi Kumbhar, Venkatesh Mishra, Kevin Coutinho, Divij Handa, Ashif Iquebal, Chitta Baral (2025). North American Chapter of the Association for Computational Linguistics.