Procedural Hypernetworks: Joint Meta-Learning of Optimizers, Initializations, and Losses via Task-Specific Program Synthesis

Building on the Neural Procedural Bias Meta-Learning (NPBML) idea (Raymond et al., 2024), why not use a hypernetwork to generate each component of the learning process—optimizer, initialization, and loss—tailored to the specific task at hand? Rather than training these elements independently, the hypernetwork could take a “task embedding” and output a “learning program,” i.e., a set of weights and rules (possibly as small neural modules) governing all three components. This holistic approach has not been explored: NPBML learns procedural biases, but doesn’t use hypernetworks as generators, and Leblanc et al. (2024) use hypernetworks for parameter generation, not for the full learning pipeline. This could unlock new levels of task adaptation and help meta-learners excel in truly novel environments.

References:

1. Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks. Benjamin Leblanc, Mathieu Bazinet, Nathaniel D'Amours, Alexandre Drouin, Pascal Germain (2024).
2. Meta-Learning Neural Procedural Biases. Christian Raymond, Qi Chen, Bing Xue, Mengjie Zhan (2024). arXiv.org.