Residual Attribution Datasets: Benchmarking Depth-wise Aggregation Mechanisms

TL;DR: What if we built datasets specifically designed to stress-test how well different residual aggregation mechanisms preserve and utilize information from different depths? This idea involves assembling or synthesizing datasets and benchmarks to empirically evaluate the selective use of earlier representations, especially in challenging compositional or long-range reasoning tasks.

Research Question: Can specialized datasets and benchmarks reveal meaningful differences between fixed, attention-based, and sparse residual aggregation mechanisms in terms of information preservation and utility across depth?

Hypothesis: Novel datasets that require models to integrate information from specific depths or time steps will expose strengths and weaknesses of various residual connection strategies, guiding further innovation and tuning.

Experiment Plan: Design synthetic and real-world benchmarks where solution requires integrating information from specific layers (e.g., tasks with compositional reasoning, long-term dependencies, or deliberately obfuscated intermediate states). Train models with fixed, AttnRes, Block AttnRes, and sparse residuals. Use attribution methods to measure which layers are actually contributing to the output. Publish datasets and benchmarking protocols for the community.

References:

• Chen, K. et al. (2026). Attention Residuals.
• Ying, W., Li, J., Wu, Y., Zheng, K., Deng, Y., & Li, J. (2020). U-Net with Dense Encoder, Residual Decoder and Depth-wise Skip Connections. IEEE International Joint Conference on Neural Network.
• Menta, T. R., Agrawal, S., & Agarwal, C. (2025). Analyzing Memorization in Large Language Models through the Lens of Model Attribution. North American Chapter of the Association for Computational Linguistics.