Reward Shaping for Selective Prediction: A Comparative Study in Collaborative Self-Play

Eisenstein et al. show group reward induces meta-knowledge, but what reward shapes work best? Building on Hasan & Niyogi’s (2024) comparative analysis of reward specifications under sparsity, we propose a controlled study of multi-objective rewards (accuracy, abstention cost, tool-call cost, verification bonuses) across environments with varying signal sparsity and tool quality. The contribution is a theory-backed characterization of reward designs that encourage truthful uncertainty reporting and efficient tool use. We also borrow curriculum ideas from Pommerman self-play (Huynh et al., 2024) to anneal costs over training, revealing phase transitions in meta-knowledge emergence. The outcome would be practical recipes for reward design that boost solo agent calibration and cost-aware tool orchestration.

References:

1. Don't lie to your friends: Learning what you know from collaborative self-play. Jacob Eisenstein, Reza Aghajani, Adam Fisch, Dheeru Dua, Fantine Huot, Mirella Lapata, Vicky Zayats, Jonathan Berant (2025). arXiv.org.
2. Reward Specifications in Collaborative Multi-agent Learning: A Comparative Study. Maram Hasan, R. Niyogi (2024). ACM Symposium on Applied Computing.
3. Multi-Agent Training for Pommerman: Curriculum Learning and Population-based Self-Play Approach. Nhat-Minh Huynh, Hoang-Giang Cao, I-Chen Wu (2024). arXiv.org.