Outlier-Driven Meta-Learning: Discovering New Meta-Optimization Strategies from Unexpected Training Trajectories

Sobotka et al. (2023) point out that learned optimizers can behave unpredictably, with issues in stability and generalization. Instead of treating such outlier runs as failures, what if we mine them for insight? By collecting, clustering, and analyzing “strange” training trajectories (e.g., abrupt loss spikes, optimizer state divergence), we might uncover unrecognized patterns—like periodicity, mode switching, or anomalous state transitions—that suggest new optimizer update rules or architectures. This “outlier mining” approach could be formalized with statistical and visualization tools, then used to meta-learn or hand-design new optimizers that explicitly account for or exploit these observed anomalies. It’s a novel deviation from current literature, which typically discards or regularizes away such behaviors.

References:

1. Investigation into the Training Dynamics of Learned Optimizers. Jan Sobotka, Petr Simánek, Daniel Vasata (2023). International Conference on Agents and Artificial Intelligence.