Token-Level Meta-Learning for Efficient Continual Learning in Large Models

Seo et al.'s Train-Attention shows that token-level weighting improves continual learning efficiency in LLMs. This research synthesizes their approach with Zou & Lin's (2022) Taylor expansion approximation to create a token-aware meta-learning framework. The system would meta-learn to predict token importance scores, then apply computationally efficient Taylor-based updates only to high-impact tokens during continual learning. This addresses the conflict between efficiency (Zou & Lin) and effectiveness (Seo et al.) in current meta-CL methods. By focusing updates on critical tokens, we could achieve the computational benefits of Taylor expansion while maintaining the performance of attention-based methods—particularly valuable for large models where SAPT's (Zhao et al., 2024) parameter-efficient tuning still struggles with scalability.

References:

1. Efficient Meta-Learning for Continual Learning with Taylor Expansion Approximation. Xiaohan Zou, Tong Lin (2022). IEEE International Joint Conference on Neural Network.
2. Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learning. Yeongbin Seo, Dongha Lee, Jinyoung Yeo (2024). Neural Information Processing Systems.
3. SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models. Weixiang Zhao, Shilong Wang, Yulin Hu, Yanyan Zhao, Bing Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che (2024). Annual Meeting of the Association for Computational Linguistics.