Multimodal Interestingness: Integrating Visual and Narrative Representations to Enhance LLM Alignment with Human Judgments

Research Question: Does providing LLMs with multimodal (text + visual) representations of math problems improve their alignment with human interestingness judgments?

Hypothesis: Access to visual and narrative elements will enable LLMs to better mirror the nuanced factors driving human interestingness, reducing the distributional gap observed in prior studies.

Experiment Plan: Select a set of math problems with associated diagrams/narratives. Have humans rate the interestingness of each in different modalities. Fine-tune or prompt LLMs with the same representations, collecting their ratings. Analyze alignment and rationales across modalities, and compare shifts in agreement.

References: 1. Mishra, S., et al. (2025). A Matter of Interest: Understanding Interestingness of Math Problems in Humans and Language Models.
2. Ahn, D., Choi, Y., Yu, Y., Kang, D., & Choi, J. (2024). Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback. Annual Meeting of the Association for Computational Linguistics.