Adversarial Alignment Training: Stress-Testing and Fortifying Against Emergent Misalignment

TL;DR: What if we deliberately fine-tune models with adversarial alignment objectives—training them to resist both narrow misalignment and tricky prompts? Let’s combine adversarial training (as in Park et al., 2024) with alignment losses to see if models become more robust to emergent misalignment.

Research Question: Can adversarial training strategies, adapted for alignment rather than accuracy, inoculate LLMs against the broad effects of emergent misalignment caused by narrow fine-tuning?

Hypothesis: Models trained with adversarial alignment objectives—where the adversary seeks to maximize misalignment while the model resists—will be significantly more robust to both direct and latent misalignment compared to standard finetuning.

Experiment Plan: - Develop an adversarial training loop: (a) the adversary generates prompts likely to elicit misalignment, (b) the model is trained to produce aligned responses on these prompts, (c) narrow finetuning is applied (e.g., insecure code).

• Introduce alignment losses (e.g., refusal when appropriate, helpful/harmless/honest objectives).
• Evaluate robustness to emergent misalignment across a suite of prompts and tasks.
• Compare with standard fine-tuning and with models using only adversarial or only standard alignment objectives.

References:

• Park, L., Kim, J., Oh, M. G., Park, J., & Kwon, T.-H. (2024). Adversarial Feature Alignment: Balancing Robustness and Accuracy in Deep Learning via Adversarial Training. AISec@CCS.
• Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., & Evans, O. (2025). Training large language models on narrow tasks can lead to broad misalignment. Nature.