Longitudinal Change-Aware Multi-Modal Fusion with State-Space Modeling

Remote sensing has a rich literature on change detection via multi-modal fusion (Saidi et al. 2024 review). Biomedical imaging needs a comparable paradigm for therapy response and disease progression (e.g., tumor shrinkage on MRI/CT, metabolic changes on PET, evolving atrophy patterns in AD). We propose a longitudinal, Siamese multi-modal architecture with a light, long-context state-space backbone inspired by MFMamba (Wang et al. 2024): one branch processes imaging sequences (e.g., MRI/PET across visits) with CNN/Vision-Transformer encoders feeding into a Mamba module for temporal dependencies; a parallel branch handles clinical tables/omics (TriFormer/MetaFusion style), also aggregated via state-space dynamics.

We add two key components: (1) cross-time cross-modal attention to align modalities across visits and detect change patterns that are modality-consistent (e.g., structural and metabolic changes reinforcing each other), and (2) a change-detection loss that supervises both voxel-level delta maps and patient-level outcomes, transferring ideas from remote sensing Siamese fusion. To deal with registration error and sensor variability, incorporate feature-based registration (Shojaei et al. 2025) within the training loop and evidential uncertainty (Tang and Zhu 2025) to propagate alignment uncertainty into predictions. FiHam’s progressive modal-aware fusion (Lu et al. 2025) can be adapted to segregate modality-specific features at lower levels and integrate them at higher temporal levels.

This differs from static fusion models and from single-timepoint AD classifiers (e.g., Sharma et al. 2024). It unifies longitudinal dynamics, multi-modal fusion, and uncertainty-aware change detection in a low-compute temporal backbone. Impact-wise, it could deliver reliable, interpretable progression markers and early response indicators, with clinical utility in oncology, neurology, and cardiology.

References:

1. Novel feature-based method for multi-modal biomedical image registration compared to intensity-based technique. M. Shojaei, Lichen Yang, Kazem Shojaei, Jeerapat Doungchawee, R. Vachet (2025). Scientific Reports.
2. RAE-Net: a multi-modal neural network based on feature fusion and evidential deep learning algorithm in predicting breast cancer subtypes on DCE-MRI. Xiaowen Tang, Yinsu Zhu (2025). Biomedical engineering and physics express.
3. TriFormer: A Multi-modal Transformer Framework For Mild Cognitive Impairment Conversion Prediction. Linfeng Liu, Junyan Lyu, Siyu Liu, Xiaoying Tang, S. Chandra, F. Nasrallah (2023). IEEE International Symposium on Biomedical Imaging.
4. Metafusion: A Novel Method for Integrating Clinical Metadata with Imaging Modalities for Medical Applications. A. Raghu, Anisha Raghu (2025). IEEE International Symposium on Biomedical Imaging.
5. Fine-Grained Hierarchical Progressive Modal-Aware Network for Brain Tumor Segmentation.. Chenggang Lu, Jianwei Zhang, Dan Zhang, Lei Mou, Jinli Yuan, Kewen Xia, Zhitao Guo, Jiong Zhang (2025). IEEE journal of biomedical and health informatics.
6. Deep-Learning for Change Detection Using Multi-Modal Fusion of Remote Sensing Images: A Review. Souad Saidi, Soufiane Idbraim, Younes Karmoude, Antoine Masse, M. Arbelo (2024). Remote Sensing.
7. Deep convolutional neural network framework with multi-modal fusion for Alzheimer’s detection. M. Sharma, M. Kaiser, K. Ray (2024). International Journal of Reconfigurable and Embedded Systems (IJRES).
8. MFMamba: A Mamba-Based Multi-Modal Fusion Network for Semantic Segmentation of Remote Sensing Images. Yan Wang, Li Cao, He Deng (2024). Italian National Conference on Sensors.