Residual-LM Consistency Probes for Multi-Modal Biomedical Fusion

Lai et al. (CVPRW 2024) showed an unexpected finding: residual-based LLM blocks, used as frozen transformer layers, can directly process visual tokens and boost purely visual biomedical tasks without any text. Building on this surprising modality-agnostic inductive bias, this idea treats LLM residual blocks as plug-and-play “consistency probes” inside a multi-modal fusion network. Instead of only using them to encode single images, we route modality-specific tokens (e.g., MRI, PET, CT, clinical tabular) through shared frozen LLM residuals and explicitly model cross-modal residuals as signals of disagreement. Cases where modalities disagree tend to be the ones that hurt performance in practice—e.g., misregistration, contrast timing issues, or functional-structural mismatches—and also the ones we want to flag.

Concretely, we: (1) fuse with a late-fusion backbone (motivated by Upadhya et al. 2024 on MIMIC-CXR, where late fusion outperformed early fusion), (2) inject one or more frozen LLM residual blocks as a shared token mixer across modalities to generate modality-consistency scores, and (3) use an evidential uncertainty head (as in RAE-Net; Tang and Zhu 2025) to modulate predictions based on detected inconsistencies. The architecture can sit atop specialized modality encoders like TriFormer’s image/clinical branches (Liu et al. 2023), or MetaFusion (Raghu and Raghu 2025) for clinical metadata integration.

What’s novel here is treating LLM residuals not just as encoders but as cross-modal consistency detectors—making the “unexpected” finding of Lai et al. operational for fusion. The potential impact is twofold: better robustness (by down-weighting unreliable modalities when they conflict) and clinically meaningful anomaly flags on discordant cases (e.g., PET-avid lesions without MRI correlate), which are exactly the cases clinicians scrutinize.

References:

1. Residual-based Language Models are Free Boosters for Biomedical Imaging Tasks. Zhixin Lai, Jing Wu, Suiyao Chen, Yucheng Zhou, N. Hovakimyan (2024). 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).
2. RAE-Net: a multi-modal neural network based on feature fusion and evidential deep learning algorithm in predicting breast cancer subtypes on DCE-MRI. Xiaowen Tang, Yinsu Zhu (2025). Biomedical engineering and physics express.
3. TriFormer: A Multi-modal Transformer Framework For Mild Cognitive Impairment Conversion Prediction. Linfeng Liu, Junyan Lyu, Siyu Liu, Xiaoying Tang, S. Chandra, F. Nasrallah (2023). IEEE International Symposium on Biomedical Imaging.
4. Metafusion: A Novel Method for Integrating Clinical Metadata with Imaging Modalities for Medical Applications. A. Raghu, Anisha Raghu (2025). IEEE International Symposium on Biomedical Imaging.
5. Advancing Medical Image Diagnostics through Multi-Modal Fusion: Insights from MIMIC Chest X-Ray Dataset Analysis. J. Upadhya, K. Poudel, J. Ranganathan (2024). International Conference on Multimodal Interaction.