Unknown-Aware Embodied Planning: Fusing Negative-Label OOD Signals with VLM-Guided Robot Control

Add an OOD gate to VLM-based embodied controllers (e.g., VLM-PC for legged robots; Chen et al., 2024; VLM-Social-Nav; Song et al., 2024) that continuously scores camera frames using negative-label OOD detection (NegLabel; Jiang et al., 2024) and automated prompt tuning (LAPT; Zhang et al., 2024). High OOD scores trigger safe fallback skills, replanning, or active information gathering (e.g., repositioning to reduce uncertainty), and dynamically prompt the VLM with “what it is not” to refine plans. OOD detection in VLMs has been text-aware (NegLabel/LAPT) but rarely closed-loop in embodied planning. This marries text-derived negatives and automatic prompt tuning to real-time behavior selection, creating a language-aware “unknowns” interface that modulates the skill planner. Builds directly on the planning-and-replanning structure in VLM-PC and the social cost scoring in VLM-Social-Nav, replacing fixed priors with adaptive OOD-informed costs. Uses NegLabel’s negative labels and LAPT’s autonomous prompt optimization for robust OOD scoring under domain shift. Embodied errors often happen in the long tail of “unknown unknowns.” Treating textual negatives as first-class planning signals should reduce unsafe or socially inappropriate actions without heavy human supervision. More robust, socially compliant, and fail-safe navigation and locomotion in novel environments—critical for search-and-rescue, indoor assistance, and crowded public spaces.

References:

1. Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models. Annie S. Chen, Alec M. Lessing, Andy Tang, Govind Chada, Laura M. Smith, Sergey Levine, Chelsea Finn (2024). IEEE International Conference on Robotics and Automation.
2. Negative Label Guided OOD Detection with Pretrained Vision-Language Models. Xue Jiang, Feng Liu, Zhengfeng Fang, Hong Chen, Tongliang Liu, Feng Zheng, Bo Han (2024). International Conference on Learning Representations.
3. LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models. Yabin Zhang, Wen-Qing Zhu, Chenhang He, Lei Zhang (2024). European Conference on Computer Vision.
4. VLM-Social-Nav: Socially Aware Robot Navigation Through Scoring Using Vision-Language Models. Daeun Song, Jing Liang, Amirreza Payandeh, Amir Hossain Raj, Xuesu Xiao, Dinesh Manocha (2024). IEEE Robotics and Automation Letters.