Multi-Modal Self-Play with High-Cost Vision Experts: When to Pay for Perception

VipAct (Zhang et al., 2024) shows that orchestrating specialized vision agents boosts fine-grained perception. We use Eisenstein et al.’s group reward framing to learn cost-aware calling policies: agents are penalized for invoking high-cost vision tools unnecessarily and for confident-but-wrong answers when they should have called an expert. The novelty is treating multi-modal expert invocation as selective prediction over information modalities, not just over textual tools. We test on visual question answering and perception-heavy tasks with cost heterogeneity. Expected impact: VLMs that generalize better to long-tail visual phenomena by strategically escalating to pixel-precise tools only when necessary.

References:

1. Don't lie to your friends: Learning what you know from collaborative self-play. Jacob Eisenstein, Reza Aghajani, Adam Fisch, Dheeru Dua, Fantine Huot, Mirella Lapata, Vicky Zayats, Jonathan Berant (2025). arXiv.org.
2. VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use. Zhehao Zhang, Ryan A. Rossi, Tong Yu, Franck Dernoncourt, Ruiyi Zhang, Jiuxiang Gu, Sungchul Kim, Xiang Chen, Zichao Wang, Nedim Lipka (2024). arXiv.org.