Cross-Modal Evil Twins: Extending Obfuscated Prompts to Vision-Language Models

TL;DR: Can we create “evil twin” prompts for multimodal models like CLIP—where both text and images are obfuscated but still guide the model to the same outcome?

Research Question: Is it possible to construct cross-modal “evil twins” for vision-language models that preserve downstream behavior while being unintelligible to both humans and conventional image classifiers?

Hypothesis: With appropriate optimization, both text and image prompts can be obfuscated to a degree that they are unrecognizable to humans, yet still elicit equivalent classification or retrieval behavior from VLMs.

Experiment Plan: - Use methods akin to LAPT (Zhang et al., 2024) and CoOp (Zhou et al., 2021) to generate obfuscated textual and visual prompts.

• Evaluate model output similarity (e.g., class probability distributions) between original and obfuscated prompt pairs.
• Measure human interpretability via crowdsourcing and test transferability across different VLM architectures.
• Analyze failure cases to understand modality-specific constraints.

References:

• Zhang, Y., Zhu, W.-Q., He, C., & Zhang, L. (2024). LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models. European Conference on Computer Vision.
• Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2021). Learning to Prompt for Vision-Language Models. International Journal of Computer Vision.
• Cao, Y., Xu, X., Sun, C., Cheng, Y., Du, Z., Gao, L., & Shen, W. (2023). Personalizing Vision-Language Models With Hybrid Prompts for Zero-Shot Anomaly Detection. IEEE Transactions on Cybernetics.