Causal Benchmark Suite: Annotated Datasets for Coordination Error Propagation

TL;DR: Let’s make agent system science easier by building “causal benchmark” datasets—every agent’s action is time-stamped and causally tagged, so we can trace how errors ripple through the system. We’ll release these datasets for the community to accelerate research on coordination failures and debugging.

Research Question: How can richly annotated multi-agent datasets—with timestamps, causal ties, and delta tracking—enable new methods for diagnosing, predicting, and mitigating coordination failures?

Hypothesis: Datasets with granular annotations (action, timestamp, causal parent, delta) will facilitate the development of automated tools for early detection of coordination breakdowns, and will improve reproducibility and benchmarking for agent system research.

Experiment Plan: Design and release a suite of annotated datasets across common agent system benchmarks, each with full causal and temporal logs. Develop baseline diagnostic tools (e.g., causal graph visualization, trigger detection). Evaluate the utility of these datasets by inviting external groups to build and benchmark new coordination error detection and mitigation algorithms. Track adoption and improvements in error diagnosis/mitigation over time.

References:

• Gyevnar, B., Wang, C., Lucas, C. G., Cohen, S. B., & Albrecht, S. V. (2023). Causal Explanations for Sequential Decision-Making in Multi-Agent Systems. Adaptive Agents and Multi-Agent Systems.
• Wang, Q., Wang, T., Tang, Z., Li, Q., Chen, N., Liang, J., & He, B. (2024). MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPs. Annual Meeting of the Association for Computational Linguistics.
• Yuan, Y., He, W., Du, W., Tian, Y.-C., Han, Q.-L., & Qian, F. (2024). Distributed Gradient Tracking for Differentially Private Multi-Agent Optimization With a Dynamic Event-Triggered Mechanism. IEEE Transactions on Systems, Man, and Cybernetics: Systems.