Theory-Grounded Hypernetworks: PAC-Bayes Guided Architecture Search for Meta-Learned Optimizers

Leblanc et al. (2024) leverage PAC-Bayes and sample compression for generalization bounds in hypernetwork meta-learning, but focus on model weights. Inspired by this, we could apply similar theoretical frameworks to the meta-learning of optimizer architectures themselves. A hypernetwork could encode an entire space of possible optimizer architectures (e.g., recurrent nets, feedforward nets, hybrids) and, informed by PAC-Bayes-derived risk bounds, output optimizer architectures that are theoretically likely to generalize well across tasks. This would be a major conceptual reconfiguration: instead of searching for model weights or hyperparameters, we’re searching for the “shape” and “rules” of the optimizer itself, with statistical guarantees. This could bring much-needed rigor to the wild west of learned optimizer design, and help bridge the gap between empirical and theoretically principled meta-learning.

References:

1. Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks. Benjamin Leblanc, Mathieu Bazinet, Nathaniel D'Amours, Alexandre Drouin, Pascal Germain (2024).
2. Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks. Benjamin Leblanc, Mathieu Bazinet, Nathaniel D'Amours, Alexandre Drouin, Pascal Germain (2024).