Behavioral Experimental Paradigms for Large Language Models: A Methodological Framework for "Model Psychology"

This research idea proposes creating a systematic research program that adapts classic behavioral science paradigms into prompting protocols for large language models (LLMs). The goal is to move beyond ad-hoc prompting and establish standardized, reproducible experimental protocols that treat LLMs as behavioral study subjects. Examples include translating cognitive bias tests (e.g., the Linda problem, anchoring effects) into prompts to reveal reasoning patterns, designing preference elicitation experiments inspired by behavioral economics, and creating LLM equivalents of experiments studying attention, memory, and decision-making under uncertainty. The approach aims to bring the rigor of experimental psychology to LLM study, enabling shared behavioral assays across research groups and models. Furthermore, it envisions integrating these behavioral experiments with mechanistic interpretability methods to link observed behaviors to internal model representations, thus bridging behavioral and mechanistic understanding. Extending this framework could also systematically study model preferences and trade-offs across values like accuracy, safety, and creativity. Ultimately, this would establish "model psychology" as a rigorous scientific discipline, providing standardized tools for comparing models, tracking behavioral changes with fine-tuning, and developing theories of LLM cognition, operationalizing the vision of treating prompting as behavioral science rather than engineering.

References:

1. A tutorial on open-source large language models for behavioral science. Zak Hussain, Marcel Binz, Rui Mata, Dirk U. Wulff (2024). Behavior Research Methods.
2. Prompting as Scientific Inquiry. Ari Holtzman, Chenhao Tan (2025). arXiv.org.
3. Mechanistic Interpretability as Statistical Estimation: A Variance Analysis of EAP-IG. Maxime M'eloux, Franccois Portet, Maxime Peyrard (2025).
4. Using cognitive models to reveal value trade-offs in language models. Sonia K. Murthy, Rosie Zhao, Jennifer Hu, S. Kakade, Markus Wulfmeier, Peng Qian, Tomer D. Ullman (2025).