Conflict-Driven Hypothesis Generation: Mining Contradictory Prompt Outcomes for Theory Building

Many works (e.g., Yu et al., 2024; Mu et al., 2025; Ying et al., 2023) report conflicting findings—for example, about how models handle prompt ambiguity or system/user prompt clashes. Instead of treating these as noise, this research formalizes a pipeline to extract, categorize, and synthesize these contradictions. Drawing on meta-analytic techniques and qualitative synthesis, the goal is to build new theoretical models—perhaps even typologies—of LLM prompt robustness, capturing when and why models “flip” behavior under seemingly similar conditions. The resulting theories could explain observed brittleness, suggest new evaluation protocols, and guide robust model design. This approach explicitly leverages conflict as a creative engine for theory-building, which is underutilized in prompt science.

References:

1. SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation. Jieming Yu, An-Chi Wang, Wenzhen Dong, Mengya Xu, Mobarak Islam Hoque, Jie Wang, Long Bai, Hongliang Ren (2024). arXiv.org.
2. A Closer Look at System Prompt Robustness. Norman Mu, Jonathan Lu, Michael Lavery, David Wagner (2025). arXiv.org.
3. Intuitive or Dependent? Investigating LLMs' Robustness to Conflicting Prompts. Jiahao Ying, Yixin Cao, Kai Xiong, Yidong He, Long Cui, Yongbin Liu (2023). arXiv.org.