Multimodal Hypothesis Generation: Integrating Visual and Textual Data in LLM-driven Discovery

Most state-of-the-art work (e.g., Song Tong et al., 2024; Zhou et al., 2024) bases hypothesis generation solely on textual data or knowledge graphs derived from papers. However, scientific insights often come from patterns in visuals—think of biology, medicine, or materials science. Building on recent advances in multimodal AI (Raunak et al., 2019), this project would develop a system where LLMs are combined with vision models to jointly process paper figures, tables, and text, generating hypotheses that link trends seen in images with textual claims or gaps. For example, the model might detect an anomaly in a microscopy image and suggest a novel molecular mechanism, or correlate a pattern in a figure with an unexplored causal factor from the text. This could be evaluated for novelty and utility using benchmarks like IdeaBench, possibly leading to richer, more integrated scientific discoveries.

References:

1. Automating psychological hypothesis generation with AI: when large language models meet causal graph. Song Tong, Kai Mao, Zhen Huang, Yukun Zhao, Kaiping Peng (2024). Humanities and Social Sciences Communications.
2. IdeaBench: Benchmarking Large Language Models for Research Idea Generation. Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Eric Xie, Stefan Bekiranov, Aidong Zhang (2024). arXiv.org.
3. Hypothesis Generation with Large Language Models. Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, Chenhao Tan (2024). NLP4SCIENCE.
4. On Leveraging the Visual Modality for Neural Machine Translation. Vikas Raunak, Sang Keun Choe, Quanyang Lu, Yi Xu, Florian Metze (2019). International Conference on Natural Language Generation.