Disentangling the Origins of Scaling Asymmetry in Multimodal Pretraining

TL;DR: Why does vision need so much more data than language in multimodal models? Let’s take existing scaling laws, systematically manipulate data distributions, and run targeted ablations to see what really drives this asymmetry. For example, what happens if we level the playing field by controlling for content complexity and redundancy in vision vs. language datasets?

Research Question: What are the fundamental sources of the observed scaling asymmetry between vision and language modalities in multimodal pretraining, and can they be mitigated through dataset design or architectural changes?

Hypothesis: The scaling asymmetry arises not just from inherent modality complexity, but also from differences in dataset entropy, redundancy, and information density; controlling these factors will reduce or reshape the observed asymmetry.

Experiment Plan: Curate synthetic and natural datasets for both vision and language, matched for entropy and redundancy (e.g., using compressibility metrics). Pretrain multimodal models with the Transfusion framework and MoE, varying only data characteristics. Measure learning curves, sample efficiency, and downstream performance as data quantity scales for each modality. Analyze whether scaling asymmetry persists, is mitigated, or reverses under these controlled conditions. Extend to real-world datasets by subsampling or augmenting them to match information-theoretic properties.

References:

• Shengbang Tong, D. Fan, J. Nguyen, E. Brown, G. Zhou, S. Qian, B. Zheng, T. Vallaeys, J. Han, R. Fergus, N. Murray, M. Ghazvininejad, M. Lewis, N. Ballas, A. Bar, M. Rabbat, J. Verbeek, L. S. Zettlemoyer, K. Sinha, Y. LeCun, S. Xie. (2026). Beyond Language Modeling: An Exploration of Multimodal Pretraining.
• Enyi Shi, P. Shao, Y. Zhang, C. Cui, J. Lyu, X. Xie, X. Xia, F. Shen, T.-S. Chua. (2026). Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models. arXiv.org.
• Heng Zhang, H. Hu, Y. Shen, W. Yu, Y. Yuan, H. You, G. Cheng, Z. Zhang, L. Gan, H. Wei, H. Zhang, J. Huang. (2025). AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models. arXiv.org.