Attention Head Specialization Lottery: When Winning Tickets Learn Different Tricks

TL;DR: We're investigating whether different attention heads in winning lottery tickets specialize in different tasks, like some focusing on syntax while others handle semantics. By analyzing head-specific performance across diverse NLP tasks, we expect to discover that SLTs naturally emerge with functional specialization among heads.

Research Question: Do strong lottery tickets in MHAs exhibit emergent head specialization, and how does this compare to dense transformer architectures?

Hypothesis: SLTs in MHAs develop more pronounced head specialization than dense models, with individual heads focusing on distinct linguistic phenomena (syntax, semantics, discourse).

Experiment Plan: 1. Train large transformer models on multi-task NLP datasets
2. Extract SLTs using magnitude-based pruning
3. Create head-specific probing tasks (syntactic dependency, semantic role, coreference)
4. Compare head specialization patterns between SLTs and original dense models
5. Analyze whether head removal in SLTs has more catastrophic effects than in dense models

References: 1. Otsuka, H., et al. (2025). The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms.
2. Movva, R., & Zhao, J. (2020). Dissecting Lottery Ticket Transformers: Structural and Behavioral Study of Sparse Neural Machine Translation. BlackboxNLP Workshop.