AutoRedICL: Multimodal In-Context Learning via Self-Generated Adversarial Demonstrations and Universal Embeddings

Extend recent evidence that VLMs lack strong multimodal ICL (Doveh et al., 2024) by building an automated curriculum: (i) generate challenging multimodal instances via self-red-teaming (IDEATOR; Wang et al., 2024), including tricky visual-text combinations and “near-miss” negatives; (ii) embed items with VLM2Vec (Jiang et al., 2024) to select diverse, representative few-shot demonstrations per query; (iii) instruction-tune for ICL following Doveh et al.’s multi-turn curriculum; and (iv) add adversarial prompt tuning (AdvPT; Zhang et al., 2023) to harden against image-space perturbations and improve transfer robustness. Instead of hand-curated demos, the model synthesizes and curates its own adversarial demonstrations to maximize ICL coverage by embedding diversity. It unifies red teaming, universal multimodal embeddings, and ICL-specific instruction tuning in a single training/evaluation pipeline. IDEATOR shows VLMs can red-team themselves with high transferability; VLM2Vec shows VLMs are strong universal embedders for diverse multimodal tasks; Doveh et al. present a practical curriculum for improving ICL in VLMs; AdvPT improves robustness through learnable text prompts aligned with adversarial image embeddings. ICL is a key capability for generalization without parameter updates. Automating hard example generation and selection should lead to a step-change in ICL reliability, even under adversarial or OOD conditions. More dependable few-shot multimodal performance in the wild (retrieval, VQA, grounding) and a principled pathway to stress-test and improve ICL at scale.

References:

1. IDEATOR: Jailbreaking Large Vision-Language Models Using Themselves. Ruofan Wang, Bo Wang, Xiaosen Wang, Xingjun Ma, Yu-Gang Jiang (2024). arXiv.org.
2. Towards Multimodal In-Context Learning for Vision & Language Models. Sivan Doveh, Shaked Perek, M. J. Mirza, Amit Alfassy, Assaf Arbelle, S. Ullman, Leonid Karlinsky (2024). arXiv.org.
3. VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, Wenhu Chen (2024). International Conference on Learning Representations.
4. Adversarial Prompt Tuning for Vision-Language Models. Jiaming Zhang, Xingjun Ma, Xin Wang, Lingyu Qiu, Jiaqi Wang, Yu-Gang Jiang, Jitao Sang (2023). European Conference on Computer Vision.