Geometry-Grounded Instruction Tuning with Spectral and View Prompts

IGAP (Yan et al., 2024) bridges gaps between graph pretraining and inductive fine-tuning via spectral-space prompts; NeRF-VPT (Chen et al., 2024) shows cascading view prompts improve novel-view synthesis; SPTNet (Wang et al., 2024) uses spatial prompt tuning to focus on transferable object parts. M^2PT (Wang et al., 2024) demonstrates that multimodal prompt tuning can enhance zero-shot instruction learning. We propose a unified geometry-grounded prompt layer for MLLMs: a spectral prompt that encodes topology-aware invariants (for graphs, 3D meshes, scene graphs) and a view prompt that encodes viewpoint/pose priors for images and 3D scenes. During instruction tuning, the model jointly conditions on these prompts, learning to map linguistic instructions to actions/answers that are stable across structural changes (new graphs) and viewpoint changes (new cameras). Compared to existing multimodal prompt work, the novelty is grounding instruction alignment in explicit geometric priors delivered via prompts, not just in learned multimodal features. Impact: stronger spatial/relational reasoning for robotics, AR/VR assistants, and scientific domains, with better inductive generalization to new environments and layouts.

References:

1. M^2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning. Taowen Wang, Yiyang Liu, J. Liang, Junhan Zhao, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, Lifu Huang, Qifan Wang, Dongfang Liu (2024). Conference on Empirical Methods in Natural Language Processing.
2. Inductive Graph Alignment Prompt: Bridging the Gap between Graph Pre-training and Inductive Fine-tuning From Spectral Perspective. Yuchen Yan, Peiyan Zhang, Zheng Fang, Qingqing Long (2024). The Web Conference.
3. SPTNet: An Efficient Alternative Framework for Generalized Category Discovery with Spatial Prompt Tuning. Hongjun Wang, S. Vaze, Kai Han (2024). International Conference on Learning Representations.
4. NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning. Linsheng Chen, Guangrun Wang, Liuchun Yuan, Keze Wang, Ken Deng, Philip H. S. Torr (2024). AAAI Conference on Artificial Intelligence.