From Apraxia to Alignment: Neuropsychological Benchmarks and Bias-Repair for VLM Spatial Reasoning

Following Noever and Noever’s finding that 24/25 VLMs fail a basic Ponzo illusion construction task (Constructive Apraxia), design a standardized suite of neuropsychological visual-spatial tests for VLMs (e.g., Ponzo, Müller–Lyer, Shepard–Metzler mental rotation, tilted room illusion). Augment training with an auxiliary objective that compels models to explicitly produce intermediate “cognitive maps” or horizon/vanishing-line estimates while answering, inspired by explicit spatial representations that improved performance in Thinking in Space (Yang et al., 2024). Rather than treating illusions as curiosities, this frames them as diagnostic probes for inductive bias and grounding. The proposed bias-repair uses an auxiliary visual world model (e.g., parametric horizon lines, surface normals, or 2D occupancy sketches) to force disentanglement of projective cues. Introduce contrastive counterfactual supervision using fine-grained difference data (e.g., Img-Diff; Jiao et al., 2024) that teaches the model to distinguish “perceived” orientation from “geometric” orientation. This extends Noever’s apraxia analogy into a comprehensive benchmark and training protocol; leverages explicit cognitive mapping shown to help spatial distance reasoning (Yang et al., 2024). Uses contrastive differences (Img-Diff) to generate minimal pairs that isolate the failure modes. If successful, VLMs should stop “following the perspective” when instructed to draw horizontal lines, a behavior strongly tied to downstream reliability in robotics, navigation, and CAD-style reasoning. Establishes a principled pathway to repair spatial reasoning in VLMs, yielding measurable gains on visual-spatial intelligence and more trustworthy performance for embodied agents and medical/engineering annotation tools.

References:

1. Constructive Apraxia: An Unexpected Limit of Instructible Vision-Language Models and Analog for Human Cognitive Disorders. David A. Noever, S. M. Noever (2024). arXiv.org.
2. Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models. Qirui Jiao, Daoyuan Chen, Yilun Huang, Yaliang Li, Ying Shen (2024). Computer Vision and Pattern Recognition.
3. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Fei-Fei Li, Saining Xie (2024). Computer Vision and Pattern Recognition.