Theory-of-Mind-Aware Self-Play for Delegation and Deference

Eisenstein et al. rely on group incentives to teach “knowing what you know.” Li et al. (2023) show ToM deficits in LLM agents and benefits from explicit belief states. We combine these: agents maintain structured beliefs about each teammate’s parametric strengths and tool access, learned via self-play. Rewards favor accurate ToM inferences and effective delegation (e.g., deferring to the agent predicted to have relevant non-parametric knowledge). This diverges from implicit calibration by building meta-knowledge about others, not just self. The innovation is a ToM-regularized collaborative self-play that yields better long-horizon planning and less hallucinated certainty, with potential gains in long-context tasks (Audit-LLM; Song et al., 2024) where faithfulness and evidence tracking are critical.

References:

1. Don't lie to your friends: Learning what you know from collaborative self-play. Jacob Eisenstein, Reza Aghajani, Adam Fisch, Dheeru Dua, Fantine Huot, Mirella Lapata, Vicky Zayats, Jonathan Berant (2025). arXiv.org.
2. Theory of Mind for Multi-Agent Collaboration via Large Language Models. Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, Katia P. Sycara (2023). Conference on Empirical Methods in Natural Language Processing.
3. Audit-LLM: Multi-Agent Collaboration for Log-based Insider Threat Detection. Chengyu Song, Linru Ma, Jianming Zheng, Jinzhi Liao, Hongyu Kuang, Lin Yang (2024). arXiv.org.