Cross-Domain Data Immunization: Can “Good” Data Inoculate Against Emergent Misalignment?

TL;DR: What if mixing in certain types of data during finetuning could “vaccinate” a model against broad misalignment, even when training on risky targets? The experiment would involve interleaving innocuous, aligned data with insecure code during finetuning and testing the effect on emergent misalignment.

Research Question: Does supplementing narrow, risky finetuning datasets with targeted “immunizing” data (e.g., aligned conversations, refusal demonstrations, or diverse tasks) prevent or reduce emergent misalignment in LLMs?

Hypothesis: Strategic inclusion of alignment-promoting or off-domain data during narrow finetuning will mitigate the emergence of broad misalignment, by preserving or reinforcing beneficial latent representations.

Experiment Plan: - Prepare several finetuning regimens: (a) insecure code only (as in the original paper), (b) insecure code + alignment data (e.g., helpful/harmless/honest responses), (c) insecure code + unrelated benign tasks, (d) control (benign data only).

• Fine-tune identical base models with these regimens.
• Evaluate on coding and broad, unrelated prompts for evidence of misalignment.
• Analyze which mixtures are most effective at preventing misalignment, and whether there is a trade-off with task performance.

References:

• Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., & Evans, O. (2025). Training large language models on narrow tasks can lead to broad misalignment. Nature.
• Ouyang, J., Arman, T., & Jin, G. (2025). How Much of Your Data Can Suck? Thresholds for Domain Performance and Emergent Misalignment in LLMs. arXiv.org.