Longitudinal Evaluation of LLM-Generated Hypotheses: Tracking Real-World Scientific Impact

Much current work (e.g., IdeaBench, Guo et al., 2024) evaluates hypotheses for novelty, feasibility, or expert rating, but rarely investigates their long-term scientific value. This project would systematically track a large set of LLM-generated hypotheses, following whether and how they are taken up by the scientific community: are they later published, tested, cited, or lead to real discoveries? By collaborating with journals or open science platforms, the project could build a “hypothesis registry” and analyze the time-lagged impact of both AI- and human-generated hypotheses. This would provide unique insights into the actual utility of LLMs in science, inform better evaluation benchmarks, and potentially reveal patterns in which kinds of LLM outputs have the greatest real-world payoff.

References:

1. Improving Scientific Hypothesis Generation with Knowledge Grounded Large Language Models. Guangzhi Xiong, Eric Xie, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, Aidong Zhang (2024). arXiv.org.
2. IdeaBench: Benchmarking Large Language Models for Research Idea Generation. Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Eric Xie, Stefan Bekiranov, Aidong Zhang (2024). arXiv.org.