Transfusion++: Towards a Unified Diffusion-Transformer Model for All Modalities

TL;DR: What if we could use a single, generalized modeling objective for both vision and language, instead of diffusion for vision and next-token prediction for language? Let’s try fusing diffusion and transformer objectives so both text and images are processed and generated via a hybrid model.

Research Question: Can a unified modeling framework—combining diffusion and autoregressive modeling—enable more effective joint pretraining and generation across vision, language, audio, and action modalities?

Hypothesis: Integrating a hybrid diffusion-autoregressive objective across all modalities (not just vision and language) will yield more coherent cross-modal representations, facilitate transfer to new tasks, and reduce modality-specific artifacts.

Experiment Plan: Design a hybrid model that supports both autoregressive and diffusion-based prediction for all modalities (e.g., using discrete diffusion timestep tokens for vision/audio, as in Pan et al. 2025). Train on mixed-modality datasets (images, text, audio, action). Evaluate on cross-modal generation (e.g., text-to-image, audio-to-text), world modeling, and compositional reasoning benchmarks. Compare to the baseline Transfusion (diffusion for vision, next-token for text) and measure improvements in sample quality, cross-modal consistency, and representation synergy.

References:

• Shengbang Tong, D. Fan, J. Nguyen, et al. (2026). Beyond Language Modeling: An Exploration of Multimodal Pretraining.
• Kaihang Pan, W. Lin, Z. Yue, T. Ao, L. Jia, W. Zhao, J. Li, S. Tang, H. Zhang. (2025). Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens. Computer Vision and Pattern Recognition.