Curvature- and Conflict-Aware Mixture-of-Optimizers for MAML

There are conflicting findings on how much second-order information MAML really needs. K-FAC-based acceleration (Zhang et al., 2023) and trust-region variants (Occorso et al., 2022) promise better stability, while first-order variants are cheaper but can degrade. Inspired by the gradient similarity weighting in Tak and Hong (2024), this idea trains a controller that, per episode and per layer, selects among a small toolkit: FO-MAML step, K-FAC/natural gradient step, or a trust-region step with an adaptively chosen radius. Selection is driven by online signals:

• Curvature proxies (Fisher trace, K-FAC eigenvalue stats).
• Support–query gradient agreement and variance.
• Computation budget constraints.
  The meta-objective optimizes both performance and a compute cost term, turning optimizer selection into a learned, context-aware decision. This reframes the optimizer choice conflict as a learned policy problem rather than a fixed design. The expected pay-off is twofold: near-second-order stability on hard, high-curvature tasks and FO-level speed on easy ones. The framework also plays nicely with RL settings (Lotfi & Afghah, 2024) where stability and sample efficiency are critical.

References:

1. Enhancing Model Agnostic Meta-Learning via Gradient Similarity Loss. Jae-Ho Tak, Byung-Woo Hong (2024). Electronics.
2. Meta Reinforcement Learning Approach for Adaptive Resource Optimization in O-RAN. Fatemeh Lotfi, F. Afghah (2024). IEEE Wireless Communications and Networking Conference.
3. Kronecker-factored Approximate Curvature with adaptive learning rate for optimizing model-agnostic meta-learning. Ce Zhang, Xiao Yao, Changfeng Shi, Min Gu (2023). Multimedia Systems.
4. Trust Region Meta Learning for Policy Optimization. Manuel Occorso, Luca Sabbioni, A. Metelli, Marcello Restelli (2022). Meta-Knowledge Transfer @ ECML/PKDD.