Replay Diversity: Balancing Memorization and Generalization

TL;DR: Can replaying more diverse or even "difficult" pre-training data during fine-tuning prevent overfitting to the target and improve robustness? An initial test could stratify replay data by entropy or difficulty and measure its effect on out-of-distribution generalization.

Research Question: Does the diversity or difficulty level of replayed generic data during fine-tuning affect the model's ability to generalize and resist overfitting to the target domain?

Hypothesis: Incorporating high-diversity or high-difficulty generic data into replay buffers will help the model retain generalization abilities and improve performance on both in-domain and out-of-domain tasks.

Experiment Plan: - Method: Quantify replay data by entropy (as a proxy for diversity) or by difficulty (e.g., model confidence, as in Sun et al., 2025).

• Construct replay buffers of low, medium, and high diversity/difficulty.
• Fine-tune and evaluate on target and out-of-domain (OOD) benchmarks.
• Expected Outcome: Higher diversity/difficulty may yield more robust models, but with diminishing returns if replay data becomes too unrelated.

References:

• Kotha, S., & Liang, P. (2026). Replaying pre-training data improves fine-tuning.
• Sun, Y., Shen, J., Wang, Y., Chen, T., Wang, Z., Zhou, M., & Zhang, H. (2025). Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay. arXiv.org.