Ticket Transferability Across Attention Heads: Can Winning Tickets Share Their Secrets?

TL;DR: We're testing whether a winning ticket from one attention head can be adapted to work in another head, potentially saving computation. By transferring learned subnetwork patterns between heads, we expect to find that certain ticket structures are universally effective while others are head-specific.

Research Question: To what extent are strong lottery tickets transferable between different attention heads within and across transformer layers?

Hypothesis: Partial transferability exists between SLTs of similar-type attention heads (e.g., syntactic vs syntactic), but cross-type transfers (syntactic to semantic) require significant adaptation.

Experiment Plan: 1. Identify head types using clustering on attention patterns
2. Extract SLTs from representative heads of each type
3. Transfer tickets between heads of same/different types
4. Measure transfer success via approximation error and task performance
5. Develop transfer learning protocols to maximize cross-head ticket reuse

References: 1. Otsuka, H., et al. (2025). The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms.
2. Behnke, M., & Heafield, K. (2020). Losing Heads in the Lottery: Pruning Transformer Attention in Neural Machine Translation. Conference on Empirical Methods in Natural Language Processing.