Benchmarking Nested Learning: Designing Multi-Task, Multi-Level Continual Learning Datasets

TL;DR: Let’s make the ultimate playground for Nested Learning—datasets that require balancing several types of memory and optimization at once! We’ll design and release new benchmarks that force models to use multi-level optimization and hierarchical memory to succeed.

Research Question: What properties must a benchmark dataset possess to effectively evaluate the unique strengths of Nested Learning—specifically, its multi-level optimization and continuum memory capabilities—in realistic continual learning settings?

Hypothesis: Datasets that interleave tasks with varying temporal dependencies, abstraction levels, and context requirements will better differentiate NL-based models from standard architectures, highlighting their superior flexibility and memory handling.

Experiment Plan: Assemble or generate datasets that combine tasks with short- and long-term dependencies, e.g., alternating between rapid context shifts and slow, gradual drifts (inspired by Somayajula et al., 2022). Include both vision (e.g., sequential object recognition with context switches) and language (e.g., story understanding with abrupt and subtle topic changes) domains. Establish baseline results for standard continual learning models and NL-based approaches, reporting on metrics such as forward/backward transfer, memory usage, and resilience to concept drift. Release the datasets and benchmarking code for community use.

References:

• Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025). Nested Learning: The Illusion of Deep Learning Architectures.
• Somayajula, S. A., Song, L., & Xie, P. (2022). A Multi-Level Optimization Framework for End-to-End Text Augmentation. Transactions of the Association for Computational Linguistics.