Pre-norm Alternatives: Systematic Exploration of Non-standard Layer Normalization in Transformer Architectures

TL;DR: Pre-norm is standard, but is it really optimal? Let’s invent and benchmark new normalization positions (e.g., “mid-norm” or “post-residual-norm”) and see how they impact the emergence of spikes and sinks—maybe we’ll discover architectures that avoid the pitfalls entirely.

Research Question: Can alternative normalization placements and strategies prevent the emergence or co-occurrence of massive activations and attention sinks, while retaining or improving model performance?

Hypothesis: Strategically positioning normalization layers (or introducing hybrid schemes) will reduce or eliminate the emergence of pathological outlier phenomena, leading to more stable and interpretable models.

Experiment Plan: Design and implement a suite of normalization placement strategies (e.g., mid-layer, post-residual, or even token-dependent normalization). Train and evaluate models on standard language tasks and benchmarks. Statistically analyze the prevalence and characteristics of activation outliers and attention sinks under each configuration. Measure downstream effects on training stability, quantization robustness, and interpretability.

References:

• Sun, S., Canziani, A., LeCun, Y., & Zhu, J. (2026). The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks.
• Bondarenko, Y., Nagel, M., & Blankevoort, T. (2023). Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing. Neural Information Processing Systems.