Cross-Modal Hierarchical Memory Integration: Bridging Vision, Language, and Sensorimotor Abstraction

TL;DR: What if we built a memory system that learns multi-scale abstractions across vision, language, and movement—like a cognitive brain? We’ll design and test hierarchical memory networks that integrate and transfer abstractions between modalities.

Research Question: Can nested hierarchical memory architectures facilitate transferable, multi-scale abstractions across disparate modalities (e.g., vision, language, and motor control)?

Hypothesis: Networks with shared cross-modal hierarchical memory layers will learn generalized abstractions that transfer efficiently between modalities, surpassing modality-specific architectures in tasks requiring integration or adaptation.

Experiment Plan: - Setup: Extend hierarchical memory networks (e.g., MADY, Wang et al., 2021) to process multimodal input, adding modality-specific and shared abstraction layers.

• Data: Use multimodal datasets (e.g., video-text pairs, action descriptions).
• Measurement: Evaluate on transfer learning tasks (e.g., learning in one modality, testing in another), abstraction extraction, and generalization.
• Expected Outcome: Cross-modal architectures show improved abstraction transfer and generalization compared to single-modality or non-hierarchical baselines.

References:

• Wang, L., Yang, M., Li, C., Shen, Y., & Xu, R. (2021). Abstractive Text Summarization with Hierarchical Multi-scale Abstraction Modeling and Dynamic Memory. SIGIR Conference.
• Shettigar, N., Suh, C. S., & Banerjee, D. (2024). On Developing a Novel Brain-On-Chip Platform for Enhanced Control and Design of 3D Neural Circuit Informational Dynamics. Volume 5: Dynamics, Vibration, and Control.