Beyond Surprise: Synergizing Predictive Coding and Hierarchical Attention for Deep Spatial Supersensing

TL;DR: What if, instead of just using prediction error ("surprise") to segment events in video, we also let the model focus attention hierarchically—like how humans notice both big-picture changes and fine details? For a first experiment, implement a hybrid neural model that jointly optimizes next-frame prediction loss and hierarchical, self-reflective attention (as in MASR), then evaluate on VSI-SUPER.

Research Question: Can combining hierarchical attention focusing with predictive-coding-based surprise improve spatial supersensing (especially streaming event cognition and implicit 3D spatial reasoning) in long-horizon video tasks beyond what surprise alone affords?

Hypothesis: Integrating multiscale self-reflective attention, which guides the model to both coarse and fine event boundaries based on task relevance (per MASR: Shiwen Cao et al., 2025), with predictive coding’s surprise signals, will yield more robust memory management and spatial event segmentation than either mechanism in isolation.

Experiment Plan: - Setup: Develop a unified framework combining Cambrian-S’s next-latent-frame prediction with MASR's hierarchical attention modules.

• Data: Use the VSI-SUPER benchmarks and possibly EgoSchema/Video-MME for extended testing.
• Measurement: Compare event segmentation accuracy, long-term recall, and 3D spatial prediction across models: (a) predictive coding only, (b) hierarchical attention only, (c) hybrid.
• Expected Outcome: The hybrid model achieves higher performance on streaming cognition and implicit 3D tasks, showing sharper event boundaries and better 3D reasoning, supporting the hypothesis that surprise and attention are complementary cognitive drivers.

References: 1. Yang, S., Yang, J., Huang, P., Brown, E., Yang, Z., Yu, Y., Tong, S., Zheng, Z., Xu, Y., Wang, M., Lu, D., Fergus, R., LeCun, Y., Li, F., & Xie, S. (2025). Cambrian-S: Towards Spatial Supersensing in Video.
2. Cao, S., Zhang, Z., Jiao, J., Qiao, J., Song, G., Shen, R., & Meng, X. (2025). MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understanding. arXiv.org.
3. Yates, T., Yasuda, S., & Yildirim, I. (2023). Temporal segmentation and ‘look ahead’ simulation: Physical events structure visual perception of intuitive physics. bioRxiv.