Prompt Psychometrics: Establishing Causal Relationships Between Linguistic Stimuli and Neural Computations in LLMs

Building on Holtzman and Tan's claim that "prompting is behavioral science," this research operationalizes their vision by creating a formal methodology for prompt psychometrics—the systematic measurement of model responses to controlled linguistic stimuli. Unlike existing work that reveals irregular model behaviors or maps static concepts to neural features, this framework aims to establish causal links between prompt design elements and dynamic neural activations. The approach applies randomized controlled trial principles by systematically varying single prompt features while holding context constant, integrates mechanistic interpretability tools as real-time measurements during prompting experiments, and uses data from controlled experiments to train meta-models that predict neural responses to novel prompts. This transforms prompting into a scientific instrument for understanding LLM cognition and engineering reliable behaviors in safety-critical domains.

References:

1. Prompting as Scientific Inquiry. Ari Holtzman, Chenhao Tan (2025). arXiv.org.
2. Mechanistic interpretability of large language models with applications to the financial services industry. Ashkan Golgoon, Khashayar Filom, Arjun Ravi Kannan (2024). International Conference on AI in Finance.