Contradiction-Driven Benchmark Synthesis: Learning from Conflict in LLM Outputs

While works like Theorem-of-Thought (Abdaljalil et al., 2025) and Gabriel et al. (2024) highlight inconsistencies in LLM logic and differential treatment of subgroups, there is little research on proactively using these conflicts to drive evaluation. This idea tackles that head-on: systematically collect instances where LLMs’ outputs contradict themselves (across runs, models, or prompts) or diverge from human expectations—especially when these contradictions align with different logical reasoning modes (abductive, deductive, inductive) or demographic factors. The system then automatically synthesizes new hypothesis-driven benchmarks focused on these edge cases. This approach not only addresses the “conflicting results” heuristic but also ensures that evaluation remains grounded in real, high-stakes LLM failures. The resulting benchmarks would be uniquely challenging and highly diagnostic, potentially leading to more robust model improvements than static or randomly sampled test sets.

References:

1. Can AI Relate: Testing Large Language Model Response for Mental Health Support. Saadia Gabriel, Isha Puri, Xuhai Xu, Matteo Malgaroli, Marzyeh Ghassemi (2024). Conference on Empirical Methods in Natural Language Processing.
2. Theorem-of-Thought: A Multi-Agent Framework for Abductive, Deductive, and Inductive Reasoning in Language Models. Samir Abdaljalil, H. Kurban, Khalid A. Qaraqe, E. Serpedin (2025). Proceedings of the 3rd Workshop on Towards Knowledgeable Foundation Models (KnowFM).