Perturbation-Robust Legibility: Hybrid ToM Scaffolds for LLM-Powered Robots

Verma et al. (2024, HRI) show LLMs can label robot behaviors as explicable/legible/predictable/obfuscatory—but their “ToM” crumbles under trivial perturbations (inconsistent beliefs, uninformative context, conviction tests). Zhao et al. (2015, AAAI FS) argue for explicit ToM representations to track others’ beliefs. Building on these, this project proposes a hybrid architecture: a structured ToM scaffold (e.g., a probabilistic/logic model over goals, beliefs, and action costs) performs core inference, while an LLM is restricted to generating natural language rationales and clarification questions. We further incorporate domain-specific large vision models (LVMs) for perception of human/scene state where needed (Zhang et al., 2024, MobileHCI). Human-in-the-loop optimization (Slade et al., 2024, Nature) tunes the scaffold parameters for different tasks, while anxiety and expectation effects specific to embodied LLMs (Kim et al., 2024, HRI) inform how and when the robot communicates uncertainty. Novelty lies in treating the LLM not as the ToM reasoner but as a communicative layer atop a perturbation-invariant ToM core. We would evaluate robustness across the behavior types and domains in Verma et al. (2024) and extend to social navigation settings (Mavrogiannis et al., 2019, HRI). The anticipated impact is a practical recipe for making LLM-powered robots more predictable, transparent, and trustworthy when it matters—where “legibility” judgments don’t flip with prompt noise.

References:

1. Theory of Mind Abilities of Large Language Models in Human-Robot Interaction: An Illusion?. Mudit Verma, Siddhant Bhambri, Subbarao Kambhampati (2024). IEEE/ACM International Conference on Human-Robot Interaction.
2. Represent and Infer Human Theory of Mind for Human-Robot Interaction. Yibiao Zhao, Steven Holtzen, Tao Gao, Song-Chun Zhu (2015). AAAI Fall Symposia.
3. Effects of Distinct Robot Navigation Strategies on Human Behavior in a Crowded Environment. Christoforos Mavrogiannis, A. Hutchinson, J. Macdonald, Patrícia Alves-Oliveira, Ross A. Knepper (2019). IEEE/ACM International Conference on Human-Robot Interaction.
4. On human-in-the-loop optimization of human-robot interaction.. Patrick Slade, Christopher Atkeson, J. M. Donelan, H. Houdijk, Kimberly A. Ingraham, Myunghee Kim, Kyoungchul Kong, Katherine L Poggensee, Robert Riener, Martin Steinert, Juanjuan Zhang, Steve H Collins (2024). Nature.
5. Understanding Large-Language Model (LLM)-powered Human-Robot Interaction. Callie Y. Kim, Christine P. Lee, Bilge Mutlu (2024). IEEE/ACM International Conference on Human-Robot Interaction.
6. Vision Beyond Boundaries: An Initial Design Space of Domain-specific Large Vision Models in Human-robot Interaction. Yuchong Zhang, Yong Ma, Danica Kragic (2024). International Conference on Human-Computer Interaction with Mobile Devices and Services.