Universal Diffusion Model Converter: From BERTs and GPTs to DLMs, Seamlessly

TL;DR: What if you could turn any BERT or GPT model into a diffusion language model with a single command? By automating and generalizing the AR-to-DLM conversion process inside dLLM, researchers could rapidly prototype and experiment with diffusion paradigms on any pretrained model. A starter experiment would auto-convert a range of AR models (GPT-2, LLaMA) and compare resulting DLMs on language modeling benchmarks.

Research Question: How effective and generalizable is a fully automated AR-to-DLM conversion pipeline, and what are the performance trade-offs across different architectures and scales?

Hypothesis: A universal, automated conversion tool can produce competitive DLMs from diverse AR and encoder models, preserving core performance and enabling rapid exploration of diffusion-based architectures.

Experiment Plan: - Implement a one-click AR-to-DLM conversion module (drawing from Gong et al. (2024) and Ye et al. (2025)), supporting varied architectures.

• Apply to a suite of pretrained models (BERT, GPT-2, LLaMA) at multiple scales.
• Evaluate resulting DLMs on perplexity, reasoning, and fill-in-the-middle tasks, comparing to AR baselines and prior diffusion conversions.
• Identify bottlenecks or architecture-specific challenges in the conversion process.

References:

• Gong, S., Agarwal, S., Zhang, Y., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Han, J., Peng, H., & Kong, L. (2024). Scaling Diffusion Language Models via Adaptation from Autoregressive Models. International Conference on Learning Representations.
• Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., & Kong, L. (2025). Dream 7B: Diffusion Large Language Models. arXiv.org.