LayerNorm-Integrated Strong Lottery Tickets: Bridging Theory and Practice in Transformers

TL;DR: We'll explore how adding LayerNorm changes the lottery ticket game - does it help find better winning tickets or mess up the prize? By training transformers with and without LayerNorm while tracking ticket quality across layers, we expect to find that LayerNorm creates more consistent but potentially less diverse winning tickets.

Research Question: How does LayerNorm affect the existence, quality, and distribution of strong lottery tickets in multi-head attention mechanisms?

Hypothesis: LayerNorm fundamentally alters the SLT landscape in MHAs by normalizing attention distributions, leading to more stable but less diverse subnetworks compared to non-normalized transformers.

Experiment Plan: 1. Create identical transformer architectures with/without LayerNorm
2. Apply iterative magnitude pruning to find SLTs in both variants
3. Track approximation error across layers and head configurations
4. Analyze attention pattern diversity in extracted tickets using attention entropy metrics
5. Compare performance on GLUE benchmark tasks to validate theoretical predictions

References: 1. Otsuka, H., Chijiwa, D., Okoshi, Y., Fujiki, D., Takeuchi, S., & Motomura, M. (2025). The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms.
2. Wu, X., Ajorlou, A., Wang, Y., Jegelka, S., & Jadbabaie, A. (2024). On the Role of Attention Masks and LayerNorm in Transformers. Neural Information Processing Systems.