Reverse Engineering Human Interestingness: Probing LLMs with Adversarial Math Problem Rationales

Research Question: How resilient are LLMs to misleading or manipulated rationales for mathematical interestingness, compared to human judgment?

Hypothesis: LLMs will place disproportionate weight on provided rationales—even when misleading—demonstrating a lack of genuine causal understanding of mathematical interest, as compared to humans who will spot the incongruence more often.

Experiment Plan: Collect a set of math problems with genuine human rationales for interestingness. Generate adversarial rationales (e.g., by swapping rationales among problems or using GPT-4 to create plausible but incorrect rationales). Present both humans (e.g., IMO participants/crowd-workers) and LLMs with these pairings and ask for interestingness ratings. Compare sensitivity to misleading rationales between groups, and analyze response patterns for insight into reasoning mechanisms.

References: Mishra, S., Machino, Y., Poesia, G., Jiang, A., Hsu, J., Weller, A., Mishra, C., Broman, D., Tenenbaum, J., Jamnik, M., Zhang, C. E., & Collins, K. M. (2025). A Matter of Interest: Understanding Interestingness of Math Problems in Humans and Language Models.