Curriculum-Guided Self-Distillation for Robust Continual Learning in Non-Stationary Environments

TL;DR: What if we could make SDFT even better by teaching the model easy tasks first and then introducing harder ones, like how students learn in school? As a first experiment, we’d combine SDFT with an uncertainty-aware curriculum—prioritizing “clean” or less ambiguous data early on—then progressively introduce more difficult or ambiguous samples, measuring if this staged approach leads to even less forgetting and more robust skill accumulation.

Research Question: Can integrating curriculum learning principles into the SDFT framework further reduce catastrophic forgetting and improve new-task acquisition, especially in domains with significant domain shift or noisy labels?

Hypothesis: Using a curriculum—where the student model first distills from the easiest, most confident examples and gradually incorporates harder ones—will result in better stability (old skill retention) and plasticity (new skill learning) compared to standard SDFT, particularly in non-stationary or noisy environments.

Experiment Plan: Implement SDFT as per Shenfeld et al. (2026), but augment training with a Dirichlet-based uncertainty assessment (as in Tian et al., 2025). Construct a curriculum that introduces training examples from low to high uncertainty. Benchmark on sequential learning tasks with increasing domain shift or label noise (e.g., continual vision benchmarks like split CIFAR-100). Track metrics: new-task accuracy, forgetting rate, and robustness to noise. Compare against vanilla SDFT and curriculum-only baselines.

References:

• Shenfeld, I., Damani, M., Hubotter, J., & Agrawal, P. (2026). Self-Distillation Enables Continual Learning.
• Tian, F., Feng, M., Luo, J., Wu, Z., Mei, L., Yang, L., Dong, W., & Wang, Y. (2025). Generalizing to New Area: Self-Distillation Curriculum Learning for Fine-Grained Cross View Localization. ACM Multimedia.