Meta-Self-Distillation: Rapid Adaptation of Continual Learners via Model-Agnostic Meta-Learning

TL;DR: What if SDFT could learn how to learn, so it adapts quickly to totally new tasks, like a student who picks up new subjects faster because they know good study strategies? A pilot study would test whether combining SDFT with meta-learning (e.g., MAML) lets models not just remember old tasks but also rapidly adapt to new, unseen ones after just a few updates.

Research Question: How can meta-learning be integrated with SDFT to enable continual learners to rapidly adapt to new tasks or domains with minimal forgetting?

Hypothesis: A meta-learned SDFT framework (Meta-SDFT) will yield models with enhanced adaptability—able to quickly acquire new tasks with few examples—while preserving the benefits of on-policy self-distillation for knowledge retention.

Experiment Plan: Formulate SDFT as the inner-loop adaptation within a MAML-style meta-learning setup. Evaluate on standard continual learning benchmarks, introducing novel tasks mid-sequence. Measure adaptation speed (few-shot performance), long-term retention, and cross-task interference. Compare with SDFT-only, meta-learning only, and standard fine-tuning baselines.

References:

• Shenfeld, I., Damani, M., Hubotter, J., & Agrawal, P. (2026). Self-Distillation Enables Continual Learning.
• Oubagine, R., Laaouina, L., Jeghal, A., & Tairi, H. (2025). A Unified AI Architecture for Self-Regulated Learning: Cognitive Modeling, Meta-Learning, and Continual Adaptation. Algorithms.