Benchmarking Open-Ended Multi-Agent Evolution with Dynamic, Real-World-Inspired Datasets

TL;DR: To really stress-test CORAL and its descendants, let’s build new, super-diverse datasets of open-ended multi-agent problems—not just puzzles, but evolving games, dynamic medical cases, or simulated economies.

Research Question: How do autonomous multi-agent evolution frameworks like CORAL perform across a new suite of dynamic, open-ended benchmarks designed to mimic complex, real-world multi-agent interactions?

Hypothesis: Current approaches—including CORAL—will reveal specific strengths and bottlenecks when evaluated on richer, more realistic open-ended problem datasets, leading to actionable insights for framework improvements.

Experiment Plan: - Dataset Creation: Curate or synthesize datasets from platforms like Amorphous Fortress Online (Charity et al., 2025), Craftax-MA (Al Omari et al., 2025), and DynamiCare (Shang et al., 2025), emphasizing heterogeneity, persistence, and emergent behaviors.

• Benchmark Suite: Publicly release a multi-domain benchmark suite for open-ended multi-agent discovery.
• Evaluation: Run CORAL and other frameworks on these datasets, analyzing transfer learning, adaptation to new agent types, and resilience to environmental shifts.
• Expected Outcome: The new benchmarks identify capability gaps, inspire new mechanisms (e.g., for specialization, adaptation), and drive broader adoption and extension of multi-agent discovery frameworks.

References:

• Qu, A., et al. (2026). CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discovery.
• Charity, M., Wilson, M., Lee, S., Rajesh, D., Earle, S., & Togelius, J. (2025). Amorphous Fortress Online: Collaboratively Designing Open-Ended Multi-Agent AI and Game Environments. arXiv.org.
• Al Omari, B., Matthews, M., Rutherford, A., & Foerster, J. (2025). Multi-Agent Craftax: Benchmarking Open-Ended Multi-Agent Reinforcement Learning at the Hyperscale. arXiv.org.
• Shang, T., He, W., Zheng, C., Li, L., Shen, L., & Zhao, B. (2025). DynamiCare: A Dynamic Multi-Agent Framework for Interactive and Open-Ended Medical Decision-Making. arXiv.org.