Beyond Code: Executable Counterfactual Reasoning in Natural Language and Visual Domains

Building on the framework introduced by Vashishtha et al. ("Executable Counterfactuals"), which evaluates LLMs’ causal reasoning step-by-step via coded problems, this idea proposes to systematically transfer that executable counterfactual paradigm to natural language (NL) and visual reasoning tasks. The original work demonstrates a significant performance drop in LLMs when moving from simple intervention to full counterfactual reasoning, and shows that supervised finetuning struggles to generalize out-of-domain, whereas RL-based approaches help, but all of this has so far been anchored in code/math tasks.

However, the real world rarely presents problems as neatly coded scenarios. Meanwhile, as highlighted in related works—like MalAlgoQA’s diagnostic question design (Liu et al.), the systematic CLadder benchmark for natural language counterfactuals (Jin et al.), and lilGym or Flipped-VQA for RL-driven NL/visual reasoning (Wu et al., Ko et al.)—there is still a paucity of frameworks for counterfactual benchmarking in more complex, less structured domains.

The core proposal includes:

• Synthesizing and adapting the executable counterfactuals methodology to generate executable counterfactual datasets in natural language and visual reasoning domains. For NL, this involves questions requiring reconstruction of plausible “what-if” worlds and the abduction-intervention-prediction triplet, with gold-standard answers verified by causal inference tools. For vision, this involves altered scenes or video frames where counterfactual states are directly manipulable.
• Operationalizing execution not just by code run but via simulation environments or logical validation, such as generating possible world states, scoring with oracle environments, or using ‘simulated students’ in educational settings.
• Benchmarking current models on these new datasets, contrasting performance between domains and comparing code/math-trained models to those with NL and visual counterfactual exposure.
• Evaluating transferability and generalization of RL-based training strategies from code to these less formalized domains, and exploring innovations inspired by cognitive science-driven belief or memory mechanisms.

This approach aims to unify and extend executable counterfactuals beyond code, revealing whether the accuracy drop between interventional and counterfactual reasoning is general or code-specific, and to provide a multi-domain framework for understanding causal reasoning bottlenecks across modalities, fostering robust causal cognition in LLMs and multimodal models for practical real-world applications.

References:

1. MalAlgoQA: Pedagogical Evaluation of Counterfactual Reasoning in Large Language Models and Implications for AI in Education. Naiming Liu, Shashank Sonkar, Myco Le, Richard G. Baraniuk (2024). Conference on Empirical Methods in Natural Language Processing.
2. Executable Counterfactuals: Improving LLMs'Causal Reasoning Through Code. Aniket Vashishtha, Qirun Dai, Hongyuan Mei, Amit Sharma, Chenhao Tan, Hao Peng (2025).
3. lilGym: Natural Language Visual Reasoning with Reinforcement Learning. Anne Wu, Kianté Brantley, Noriyuki Kojima, Yoav Artzi (2022). Annual Meeting of the Association for Computational Linguistics.
4. CLadder: Assessing Causal Reasoning in Language Models. Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, Bernhard Scholkopf (2023).
5. Large Language Models are Temporal and Causal Reasoners for Video Question Answering. Dohwan Ko, Ji Soo Lee, Wooyoung Kang, Byungseok Roh, Hyunwoo J. Kim (2023). Conference on Empirical Methods in Natural Language Processing.