Self-Evolving Nested Optimizers: From Expressive Memory to Autonomous Learning Rule Discovery

TL;DR: Imagine if the optimizer in a neural network could learn and upgrade itself over time, just like the model it's training! Let's build a system where the optimizer is a self-modifying, nested module that adapts its own learning rules as tasks evolve.

Research Question: Can a nested, self-modifying optimizer—capable of evolving its own update rules—outperform static, hand-designed optimizers in continual learning and non-stationary environments?

Hypothesis: A meta-learned, hierarchical optimizer that leverages the "self-modifying learning module" concept from NL will adapt more effectively to changing data distributions, resulting in improved lifelong learning and faster recovery from distribution shifts compared to conventional optimizers.

Experiment Plan: Implement a meta-optimizer modeled as a nested learning module that modifies its own parameters and update strategies based on feedback from the main model's performance. Use non-stationary continual learning tasks, including language modeling on shifting corpora and time-varying vision datasets. Baseline against standard optimizers (Adam, SGD with Momentum) and recent “expressive” memory optimizers (Behrouz et al., 2025). Key measures: learning stability, catastrophic forgetting, and adaptability to new tasks. Perform qualitative analysis of learned optimizer behaviors.

References:

• Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025). Nested Learning: The Illusion of Deep Learning Architectures.
• Hao, J., Ji, K., & Liu, M. (2023). Bilevel Coreset Selection in Continual Learning: A New Formulation and Algorithm. NeurIPS.