Hybrid Tensor Product and Iterative Decomposition Attention for Enhanced Systematic Generalization in Long-Sequence Models

Building upon the groundbreaking Tensor Product Attention Is All You Need (Zhang et al., 2025), which introduced TPA for memory-efficient, scalable, and high-performing attention in long-sequence language models, this research idea proposes integrating TPA with the Attention-based Iterative Decomposition (AID) framework from Park et al. (2024). While TPA excels at compactly encoding queries, keys, and values through tensor decompositions to handle longer contexts with reduced memory overhead, it currently treats the attention process largely as a black-box compression and retrieval mechanism.

AID, on the other hand, emphasizes systematically decomposing learned tensor product representations to reveal underlying symbolic and compositional structures, addressing a known limitation in prior TPR-based models where incomplete decomposition led to poor generalization on unseen inputs.

By synthesizing these two approaches, the proposed hybrid attention mechanism would:

• Retain TPA’s compact, low-rank factorization to manage long input sequences efficiently in large transformers.
• Employ AID’s iterative and competitive attention decomposition to refine and extract compositional features from the tensor representations during the attention process.
• Enable the model not only to attend efficiently but also to explicitly disentangle and interpret structural information embedded in input sequences, which is critical for systematic generalization, reasoning, and symbolic tasks.

This novel architecture differs fundamentally from existing baselines like GQA and MLA, which focus mainly on query grouping or latent attention but do not explicitly decompose tensor representations for compositionality. It also advances beyond pure TPA by addressing its potential limitation in interpretability and symbolic structure discovery, as highlighted by Park et al.

The innovation here lies in merging TPA’s scalable tensor factorization with AID’s structured decomposition, potentially leading to models that can handle very long sequences while maintaining strong generalization to novel inputs requiring systematic reasoning. This would be particularly promising for applications in natural language understanding, code generation, and multimodal learning where both scalability and compositional generalization are essential.

If successful, this research could impact the design of next-generation transformers that are both resource-efficient and capable of more human-like compositional reasoning, pushing beyond the current state-of-the-art in both efficiency and symbolic understanding.

References:

1. Tensor Product Attention Is All You Need. Yifan Zhang, Yifeng Liu, Huizhuo Yuan, Zhen Qin, Yang Yuan, Quanquan Gu, A. C. Yao (2025). arXiv.org.
2. Attention-based Iterative Decomposition for Tensor Product Representation. Taewon Park, Inchul Choi, Minho Lee (2024). International Conference on Learning Representations.