Neural State Editors: Controlled Intervention in Internal Representations for Hierarchical RL

TL;DR: What if we could "edit" a model’s thoughts to nudge its future actions? This idea proposes directly intervening in the internal activations of an autoregressive model during RL to steer high-level behavior, akin to a teacher gently guiding a student’s internal reasoning. The core experiment would involve manipulating controller activations mid-episode to observe if and how downstream decision sequences change, testing both robustness and potential for safer or more explainable RL agents.

Research Question: Can targeted interventions in the internal representations of an autoregressive model reliably modulate the execution and composition of temporally abstract actions during hierarchical RL?

Hypothesis: Direct, structured modification of internal controller activations will allow for predictable, high-level behavioral steering, leading to more robust and controllable RL agents, especially in safety-critical or value-alignment contexts.

Experiment Plan: - Extend the internal RL framework from Kobayashi et al. (2025) with a “state editor” interface for the higher-order controller’s internal activations.

• Train models on hierarchical tasks (e.g., in grid world and MuJoCo), then intervene mid-trajectory by perturbing, resetting, or replacing controller activations.
• Measure effects on downstream behavior, task completion, and the model’s ability to recover or adapt.
• Analyze which types of interventions (e.g., semantic, random, adversarial) yield controllable vs. unpredictable outcomes.

References:

• Kobayashi, S., Schimpf, Y., Schlegel, M., Steger, A., Wolczyk, M., Oswald, J., Scherrer, N., Maile, K., Lajoie, G., Richards, B. A., Saurous, R., Manyika, J., Arcas, B. A. Y., Meulemans, A., & Sacramento, J. (2025). Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning.