Replay + Parameter-Efficient Fine-Tuning: Uncovering Synergies or Trade-offs

TL;DR: Does combining generic data replay with parameter-efficient fine-tuning (e.g., LoRA, adapters) yield additive benefits, or are there trade-offs? Try replaying generic data during LoRA-based fine-tuning and compare to standard LoRA and full fine-tuning.

Research Question: How does the interaction between generic data replay and parameter-efficient tuning methods (such as LoRA) affect fine-tuning efficiency, catastrophic forgetting, and target task adaptation?

Hypothesis: Generic data replay will enhance the effectiveness of parameter-efficient tuning methods, mitigating their tendency to overfit or forget base model capabilities, though the magnitude of synergy or interference remains an open question.

Experiment Plan: - Design: Fine-tune LLMs on target domains using (a) standard LoRA, (b) LoRA + generic replay, (c) full fine-tuning, and (d) full fine-tuning + replay.

• Metrics: Measure target and general domain performance, memory footprint, and forgetting rates.
• Data: Use Kotha & Liang (2026) datasets and methods, adapting for LoRA.
• Expected Results: Replay with LoRA should outperform standard LoRA in preserving general capabilities and possibly enhance target performance, building on findings from Riemer et al. (2025).

References:

• Kotha, S., & Liang, P. (2026). Replaying pre-training data improves fine-tuning.
• Riemer, M., Miehling, E., Liu, M., Bouneffouf, D., & Campbell, M. (2025). The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Models. arXiv.org.