Causal Anti-Prompts: Decoupled Prompting to Break Spurious Style Correlations

Amend to Alignment (Zhang et al., 2024) reduces spurious correlations in VLMs via decoupled prompt tuning; SOS-Bench (Feuer et al., 2024) shows current pipelines reward style over substance; DAPT (Cho et al., 2023) and PromptMM (Wei et al., 2024) highlight distributional and modality biases. We propose causal anti-prompts: a two-branch prompt system where one branch targets semantic competence and the other is trained adversarially to detect and suppress superficial patterns (verbosity, hedging, particular syntactic sugar) that correlate with higher judged scores but not factuality/safety. Training alternates between: (i) reinforcing semantic correctness on counterfactual, paraphrased, and OOD data; (ii) amplifying anti-prompts on curated “style traps” mined via disagreement analysis and negative-label mining (inspired by LAPT). The mechanism generalizes beyond text: in multimodal settings (M^2PT; Wang et al., 2024), anti-prompts can suppress visual artifacts and biases (cf. PROMPT-IML; Liu et al., 2024). Novelty: explicitly learning a suppressive prompt component to counteract spurious style features, rather than merely regularizing or balancing datasets. Impact: models that maintain substance under adversarial style perturbations, with improved generalization to covariate shifts and less susceptibility to judge biases.

References:

1. LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models. Yabin Zhang, Wen-Qing Zhu, Chenhang He, Lei Zhang (2024). European Conference on Computer Vision.
2. PromptMM: Multi-Modal Knowledge Distillation for Recommendation with Prompt-Tuning. Wei Wei, Jiabin Tang, Lianghao Xia, Ya Jiang, Chao Huang (2024). The Web Conference.
3. Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking. Ben Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, John P. Dickerson (2024). International Conference on Learning Representations.
4. Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models. Jie Zhang, Xiaosong Ma, Song Guo, Peng Li, Wenchao Xu, Xueyang Tang, Zicong Hong (2024). International Conference on Machine Learning.
5. PROMPT-IML: Image Manipulation Localization with Pre-trained Foundation Models Through Prompt Tuning. Xuntao Liu, Yuzhou Yang, Qichao Ying, Zhenxing Qian, Xinpeng Zhang, Sheng Li (2024). arXiv.org.
6. M^2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning. Taowen Wang, Yiyang Liu, J. Liang, Junhan Zhao, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, Lifu Huang, Qifan Wang, Dongfang Liu (2024). Conference on Empirical Methods in Natural Language Processing.
7. Distribution-Aware Prompt Tuning for Vision-Language Models. Eulrang Cho, Jooyeon Kim, Hyunwoo J. Kim (2023). IEEE International Conference on Computer Vision.