Neural “Shock Absorbers”: Designing Adaptive Normalization Layers to Decouple Massive Activations and Attention Sinks

TL;DR: What if we gave Transformers a dynamic “shock absorber” to catch extreme activations and redistribute attention before they become problematic? We’d build new normalization schemes that adapt layer-by-layer to suppress spikes and prevent persistent attention sinks, then test if this decouples the two phenomena and improves learning stability.

Research Question: Can adaptive normalization schemes, learned per layer or per head, decouple massive activations from attention sinks and improve Transformer robustness and interpretability?

Hypothesis: Introducing normalization layers that adapt based on the distribution of activations (e.g., using kurtosis or skewness metrics online) will suppress outlier spikes, reduce persistent attention sinks, and enhance both training stability and downstream performance.

Experiment Plan: Design new adaptive normalization modules (e.g., “KurtNorm” or “SpikeNorm”) that scale or clip activations based on real-time statistics. Integrate these modules into standard and pre-norm Transformers. Compare models on standard language modeling tasks, tracking the frequency and magnitude of activation outliers and sink tokens. Analyze training stability, convergence speed, and interpretability (e.g., via attribution maps). Ablation studies: selectively disable adaptive behavior to isolate causal effects.

References:

• Sun, S., Canziani, A., LeCun, Y., & Zhu, J. (2026). The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks.
• Qiu, Z., Huang, Z., Wen, K., Jin, P., Zheng, B., Zhou, Y., ... & Lin, J. (2026). A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training. arXiv.org.