Counterfactual Worldbuilding: LLM-Generated Synthetic Histories for Policy Evaluation

Building on Political-LLM’s (Li et al., 2024) proposal for counterfactual generation, this idea develops a novel "synthetic history" framework. Where Cerqua et al. (2023) require untreated units for ML Control Methods, we fine-tune LLMs to generate plausible alternative timelines (e.g., "What if the 2020 U.S. election used ranked-choice voting?"). By grounding LLM outputs in real-world institutional constraints (e.g., constitutional limits) and validating against expert judgments, we address Dahabreh & Bibbins-Domingo’s (2024) concerns about causal language in observational studies. This diverges from traditional DiD or propensity scores by creating synthetic controls for policies with no precedents, opening evaluation for radical reforms like universal basic income.

References:

1. Causal inference and policy evaluation without a control group. Augusto Cerqua, M. Letta, Fiammetta Menchetti (2023).
2. Causal Inference About the Effects of Interventions From Observational Studies in Medical Journals.. Issa J. Dahabreh, Kirsten Bibbins-Domingo (2024). Journal of the American Medical Association (JAMA).
3. Political-LLM: Large Language Models in Political Science. Lincan Li, Jiaqi Li, Catherine Chen, Fred Gui, Hongjia Yang, Chenxiao Yu, Zhengguang Wang, Jianing Cai, Junlong Aaron Zhou, Bolin Shen, Alex Qian, Weixin Chen, Zhongkai Xue, Lichao Sun, Lifang He, Hanjie Chen, Kaize Ding, Zijian Du, Fangzhou Mu, Jiaxin Pei, Jieyu Zhao, Swabha Swayamdipta, Willie Neiswanger, Hua Wei, Xiyang Hu, Shixiang Zhu, Tian-Xiang Chen, Ying-Liang Lu, Yang Shi, Li Qin, Tianfan Fu, Zhengzhong Tu, Yuzhe Yang, Jaemin Yoo, Jiaheng Zhang, Ryan A. Rossi, Liang Zhan, Liangxuan Zhao, Emilio Ferrara, Yan Liu, Furong Huang, Xiangliang Zhang, Lawrence Rothenberg, Shuiwang Ji, Philip S. Yu, Yue Zhao, Yushun Dong (2024). arXiv.org.