Sparse Attention Residuals: Learning to Skip for Efficient Deep Aggregation

TL;DR: What if instead of softly attending to all previous layers, the model could learn to "skip" or hard-select only a few relevant prior layers at each depth? This idea proposes replacing softmax aggregation in Attention Residuals with a sparse, learned selection mechanism (e.g., top-k or hard gating). An initial experiment would investigate whether this sparsity leads to better control of hidden-state growth and more interpretable layer contributions.

Research Question: Can sparse or hard-attention-based residual connections improve the efficiency, interpretability, and stability of deep language models compared to softmax-based Attention Residuals?

Hypothesis: Sparse selection mechanisms (such as top-k or learned hard gates) will reduce memory and computation further than Block AttnRes while preventing hidden-state explosion and maintaining or improving model performance. This may also enhance interpretability by highlighting which layers are truly influential at each depth.

Experiment Plan: Design a variant of Attention Residuals where each layer selects a fixed (or learned) small subset of previous layers to aggregate, using hard gating or top-k attention. Compare with standard AttnRes and Block AttnRes on benchmarks and scaling law experiments. Measure hidden-state growth, computational cost, layer contribution entropy, and downstream performance. Analyze which layers are most frequently selected to gain insight into depth-wise information flow.

References:

• Chen, K., Zhang, Y., Su, J., Xu, W., Pan, S., Wang, Y., Wang, Y., Chen, G., Yin, B., Chen, Y., Yan, J., Wei, M., Zhang, Y., Meng, F., Hong, C., Xie, X.-M., Liu, S., Lu, E., Tai, Y.-C., Chen, Y., Men, X., Guo, H., Charles, Y., Lu, H., Sui, L., Zhu, J., Zhou, Z., He, W., Huang, W., Xu, X., Wang, Y., Lai, G., Du, Y., Wu, Y., Yang, Z., & Zhou, X. (2026). Attention Residuals.
• Wang, X., Salmani, M., Omidi, P., Ren, X., Rezagholizadeh, M., & Eshaghi, A. (2024). Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models. International Joint Conference on Artificial Intelligence.