Federated Self-Distillation Fine-Tuning for Continual Learning on Edge Devices

TL;DR: Can SDFT help many edge devices (like phones or sensors) learn new things together, even if they have different data and can’t share it? As a first test, we’d create a federated version of SDFT, where each device uses self-distillation locally and periodically shares distilled knowledge (not raw data) with a central server, then measure both accuracy and communication savings.

Research Question: How can SDFT be adapted for federated continual learning scenarios, enabling heterogeneous edge devices to collaboratively acquire new skills without sharing raw data?

Hypothesis: Federated SDFT (Fed-SDFT), where each device performs local self-distillation and communicates distilled representations or pseudo-labels, will outperform traditional federated averaging and standard SDFT in terms of both knowledge retention and communication efficiency, especially with non-IID data.

Experiment Plan: Simulate a federated setting with multiple edge clients, each with distinct continual learning streams. Implement local SDFT on each client; periodically synchronize using knowledge distillation rather than model parameters. Compare Fed-SDFT to FedAvg, PerFed-SKD, and TinyFL_HKD on public datasets (e.g., MNIST, EMNIST). Key metrics: new-task accuracy, forgetting rate, communication overhead, and personalization.

References:

• Shenfeld, I., Damani, M., Hubotter, J., & Agrawal, P. (2026). Self-Distillation Enables Continual Learning.
• Singh, N., Rupchandani, J., & Adhikari, M. (2024). Personalized Federated Learning for Heterogeneous Edge Device: Self-Knowledge Distillation Approach. IEEE Transactions on Consumer Electronics.
• Hung, C.-W., Tsai, C.-Y., Wang, C.-C., & Lee, C.-H. (2025). TinyFL_HKD: Enhancing Edge AI Federated Learning With Hierarchical Knowledge Distillation Framework. IEEE Sensors Journal.