DARTBench: A Dataset for Ambiguous and Adversarial Multimodal Questions to Stress-Test Multi-Agent Tool Selection

TL;DR: Let’s build a tough new benchmark full of tricky, ambiguous, or adversarial visual questions that force DART (and other systems) to really show what they’re made of—especially in cases where disagreements are inevitable!

Research Question: Can a purpose-built dataset of ambiguous, adversarial, or multi-tool-requiring multimodal questions reveal new limitations in DART-like frameworks and spur advances in agent debate and tool recruitment strategies?

Hypothesis: Existing benchmarks under-challenge multi-agent debate frameworks; a dataset designed to maximize perceptual ambiguity, tool overlap, and adversarial distractors will expose new failure modes and highlight areas for improvement in disagreement-driven tool selection.

Experiment Plan: Curate or synthesize a dataset of VQA instances with high ambiguity, multiple plausible answers, or intentional distractors (e.g., visually similar objects, partial occlusion, adversarial image edits). Ensure each question requires multiple tools (e.g., OCR + spatial reasoning, object detection + medical knowledge). Benchmark DART, baseline multi-agent, and single-agent tool-calling methods. Analyze tool call patterns, debate trajectories, and sources of persistent disagreement. Use findings to guide new tool integrations or debate protocol enhancements. Expected outcome: The new dataset exposes gaps in current paradigms and provides a proving ground for next-gen agentic multimodal reasoning.

References:

• Sivakumaran, N., et al. (2025). DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning.
• Qiu, J., Madotto, A., Lin, Z., et al. (2024). SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM. EMNLP.
• Gai, X., Zhou, C., Liu, J., et al. (2024). MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationale. arXiv.org.