Learnable Computational Graphs: A Neural Architecture for Adaptive Long-Range Reasoning

Inspired by Bai et al.'s discovery that a successful Transformer model for multiplication builds an elegant directed acyclic graph (DAG) to cache and retrieve partial products, this research idea proposes designing a new neural architecture called the Learnable Computational Graph (LCG) module. Unlike fixed self-attention or state-space mechanisms, the LCG module dynamically constructs and executes different information flow patterns depending on the input and task. It predicts an optimal "reasoning template"—a computational blueprint such as a sparse adjacency matrix representing a DAG or hierarchical scan patterns—that is generated on the fly and differentiable. A lightweight message-passing step then routes information according to this blueprint. This approach shifts from static architectures to adaptive, learned architectures, giving the model agency to design its own computational graph for the problem at hand. The LCG module could incorporate structural regularization techniques like the "running sum" auxiliary loss from Bai et al. to encourage efficient and hierarchical computational patterns. If successful, this could lead to models that handle long-range dependencies more efficiently and interpretable reasoning processes by inspecting the generated computational graphs, advancing towards more general and flexible reasoning systems capable of switching between algorithmic strategies.

References:

1. Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls. Xiaoyan Bai, I. Pres, Yuntian Deng, Chenhao Tan, Stuart Shieber, Fernanda Vi'egas, Martin Wattenberg, Andrew Lee (2025).