Disagreement Dynamics: Uncovering Persistent Multi-Agent Disagreement to Diagnose Tool and Agent Limitations

TL;DR: What if we could use where agents keep disagreeing, even after tool calls, as a map to find where our tools or agents are weakest? By systematically analyzing persistent disagreements, we could identify “blind spots” in VLMs or tool coverage and guide future tool development or agent training.

Research Question: How can persistent multi-agent disagreements—those unresolved even after expert tool interventions—be systematically analyzed to uncover limitations in agent perception and tool coverage?

Hypothesis: Persistent disagreements will correlate with specific perceptual or reasoning blind spots within VLM agents or the available toolset, revealing actionable areas for model or tool enhancement.

Experiment Plan: Extend DART with a disagreement tracker that logs instances where agent debates remain unresolved after tool calls. Annotate these “disagreement bottlenecks” across a large, diverse VQA dataset. Cluster these cases by tool type, visual question category, and agent response patterns. Conduct error analysis to determine whether failures are due to agent misperception, tool limitations, or question ambiguity. Validate findings by augmenting the toolset or retraining agents on targeted samples and measuring reductions in persistent disagreement. Expected outcome: Clear taxonomy of disagreement sources and empirical evidence that targeted interventions reduce unresolved debates.

References:

• Sivakumaran, N., Chen, J. C.-Y., Wan, D., Zhang, Y., Yoon, J., Stengel-Eskin, E., & Bansal, M. (2025). DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning.
• Lumer, E., Nizar, F., Gulati, A., Basavaraju, P. H., & Subbiah, V. K. (2025). Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systems. arXiv.org.