Asymmetric Self-Supervised Pretraining for Heterogeneous Biomedical Modalities

Remote sensing has innovated in multi-modal SSL where modalities differ in informativeness (e.g., optical vs LiDAR). Xu et al. (2023) introduced Asymmetric Attention Fusion (AAF) with a Transfer Gate to preferentially route informative features during pretraining; Berg et al. (2023) exploited multimodal co-views for joint embeddings. Biomedical data has similar heterogeneity: structural MRI is often cleaner than PET or ASL; fundus photos vs OCT; even imaging vs clinical tables.

This idea repurposes AAF for biomedical SSL: pretrain two-stream encoders on large unlabeled datasets (e.g., multi-contrast MRI+PET, fundus+OCT, pathology+MALDI-MSI), with asymmetric attention letting the high-SNR stream guide shared representations. The Transfer Gate selectively passes fused features that improve cross-modal predictability. We can add an image-table cross-merging module (Hu et al. 2025) during pretraining to align tabular clinical covariates with imaging embeddings via cross-attention, but keep it unsupervised via masked-token and cross-view prediction tasks.

At fine-tuning, use task-specific heads such as TriFormer-style predictors (Liu et al. 2023) for MCI conversion, RAE-Net-like evidential uncertainty (Tang and Zhu 2025), or MetaFusion for clinical-imaging tasks (Raghu and Raghu 2025). The novelty is bringing asymmetric multi-modal SSL—proven in remote sensing—into biomedical fusion to reduce label burden and handle unequal modality quality. This could significantly improve data efficiency and robustness in settings where the low-SNR modality is critical but hard to learn from directly.

References:

1. RAE-Net: a multi-modal neural network based on feature fusion and evidential deep learning algorithm in predicting breast cancer subtypes on DCE-MRI. Xiaowen Tang, Yinsu Zhu (2025). Biomedical engineering and physics express.
2. TriFormer: A Multi-modal Transformer Framework For Mild Cognitive Impairment Conversion Prediction. Linfeng Liu, Junyan Lyu, Siyu Liu, Xiaoying Tang, S. Chandra, F. Nasrallah (2023). IEEE International Symposium on Biomedical Imaging.
3. Metafusion: A Novel Method for Integrating Clinical Metadata with Imaging Modalities for Medical Applications. A. Raghu, Anisha Raghu (2025). IEEE International Symposium on Biomedical Imaging.
4. Multi-Modal Fusion Network Integrating Imaging and Clinical Tabular Data for Alzheimer's Disease Classification. Wenzheng Hu, Zhenghua Guan, Nina Cheng, Ao Zhang, Yi Liu, Tianfu Wang, Baiying Lei (2025). IEEE International Symposium on Biomedical Imaging.
5. Exploring Self-Supervised Learning for Multi-Modal Remote Sensing Pre-Training via Asymmetric Attention Fusion. Guozheng Xu, Xue Jiang, Xiangtai Li, Ze Zhang, Xingzhao Liu (2023). Remote Sensing.
6. Joint Multi-Modal Self-Supervised Pre-Training in Remote Sensing: Application to Methane Source Classification. P. Berg, M. Pham, N. Courty (2023). IEEE International Geoscience and Remote Sensing Symposium.