Counterfactual Replay: Does Replaying Unrelated or Adversarial Data Help or Hurt?

TL;DR: If replaying unrelated or even adversarially chosen pre-training data during fine-tuning, does it still confer benefits, or can it degrade performance? An experiment could include negative control (irrelevant or adversarial replay) vs. standard replay.

Research Question: Is the benefit of generic data replay during fine-tuning robust to the inclusion of unrelated or adversarial data, or does it require some minimum level of relevance to the target domain?

Hypothesis: The performance gains from replay depend on the quality and relevance of the replayed data; adversarial or highly unrelated data could harm both target task learning and general capabilities.

Experiment Plan: - Setup: Define three replay conditions: (1) standard generic replay, (2) replay of unrelated domain data (e.g., replay math data when fine-tuning for poetry), and (3) adversarially selected replay (e.g., pre-training data known to contradict the target task).

• Measure: Target task performance, generalization, and model robustness.
• Expected Results: Standard replay helps; unrelated or adversarial replay either yields no benefit or actively degrades performance, clarifying boundaries of the replay effect.

References:

• Kotha, S., & Liang, P. (2026). Replaying pre-training data improves fine-tuning.
• Alajrami, A., Tan, X., & Aletras, N. (2025). Fine-Tuning on Noisy Instructions: Effects on Generalization and Performance. IJCNLP-AACL.