Path Engineering: Causal Manipulation of the "Road Not Taken"

TL;DR: We'll test whether the correlation between uncertainty and path representation is causal by artificially creating or destroying alternate paths in the model's representation space. If the core paper is right, we should see uncertainty change accordingly when we engineer the path landscape.

Research Question: Can we causally manipulate language model uncertainty by directly controlling the availability of alternate reasoning paths in the model's representation space?

Hypothesis: Artificially constraining the model's path space (by reducing the dimensionality of hidden representations along critical directions) will decrease uncertainty, while expanding the path space will increase uncertainty, even when the reasoning problem itself remains unchanged.

Experiment Plan: Identify the low-rank subspaces that encode possible reasoning paths using the methods from Dai et al. (Binding ID mechanism). Develop "path constraint" interventions that reduce activation variance along these subspace directions, effectively limiting the number of representable paths. Conversely, create "path expansion" interventions that increase variance along these directions. Apply these interventions during chain-of-thought reasoning on fixed problems and measure changes in: token-level entropy (uncertainty), ease of steering (following Zur et al.'s methodology), and final reasoning accuracy. Expected outcome: Path constraints will reduce uncertainty and make the model harder to steer, while path expansion will increase uncertainty and steerability.

References: ['Zur, A., Geiger, A., Lubana, E., & Bigelow, E.J. (2025). Are language models aware of the road not taken? Token-level uncertainty and hidden state dynamics.', 'Dai, Q., Heinzerling, B., & Inui, K. (2024). Representational Analysis of Binding in Language Models. Conference on Empirical Methods in Natural Language Processing.']