Towards Continual Spatial Supersensing: Life-Long Multi-Experience Agents with Cortical Messaging

TL;DR: If we borrow ideas from neuroscience—where brains have thousands of mini-modules that learn and communicate—could AI agents continually improve their spatial world models, even as environments change? For an initial test, create a modular memory-update system (inspired by the Thousand Brains Project), measured on continual generalization in dynamic video tasks.

Research Question: Can a modular, brain-inspired architecture—where independent "learning modules" exchange spatial and multimodal information via a messaging protocol—support continual, non-catastrophic spatial supersensing over evolving video environments?

Hypothesis: A distributed network of learning modules, each encoding and recalling different spatial or temporal subdomains, will outperform monolithic models when facing evolving or open-world video tasks, supporting continual learning and rapid adaptation as described by Clay et al. (2024) and Kang et al. (2025).

Experiment Plan: - Model: Replicate core ideas from the Thousand Brains Project and hierarchical memory update (Kang et al., 2025), integrating with Cambrian-S as the visual front-end.

• Benchmarks: Deploy on VSI-SUPER, extended with a continual adaptation regime—adding or shifting environments, tasks, or event types over time.
• Metrics: Measure transfer, generalization, and resilience to catastrophic forgetting; compare rapidity of adaptation and persistent spatial recall.
• Expected Outcome: The modular/cortical-messaging model exhibits superior continual learning, zero-shot adaptation, and preserves previous knowledge under distribution shift.

References: 1. Clay, V., Leadholm, N., & Hawkins, J. (2024). The Thousand Brains Project: A New Paradigm for Sensorimotor Intelligence. arXiv.org.
2. Kang, K., Bae, C., & Kang, D.-o. (2025). Continual Learning based on Memory Update Model for Multimodal User Modeling. The Web Conference.
3. Yang, S., Yang, J., Huang, P., Brown, E., Yang, Z., Yu, Y., Tong, S., Zheng, Z., Xu, Y., Wang, M., Lu, D., Fergus, R., LeCun, Y., Li, F., & Xie, S. (2025). Cambrian-S: Towards Spatial Supersensing in Video.