dLLM-AutoEval: Standardized, Modular Benchmarking for Diffusion LMs

TL;DR: Let's make evaluating new diffusion language models as easy as drag-and-drop—by building a standardized, extensible benchmarking suite inside dLLM that automatically tests reasoning, planning, coding, and other skills. A first step: integrate GSM8K and MATH500, plus plug-and-play support for user-contributed tasks, then compare reproducibility and coverage to existing ad-hoc evaluation scripts.

Research Question: Can an extensible, task-agnostic benchmarking suite within dLLM improve the reliability, reproducibility, and coverage of diffusion LM evaluation compared to current scattered approaches?

Hypothesis: A built-in, modular evaluation pipeline will significantly reduce evaluation friction and increase reproducibility across research groups, leading to more transparent progress tracking and fairer cross-model comparisons.

Experiment Plan: - Develop a standardized API within dLLM for adding and orchestrating benchmarks, covering math (GSM8K, MATH500), general language (HellaSwag, MMLU), and code (HumanEval).

• Automate result aggregation, statistical significance testing, and reporting.
• Invite community contributions of new benchmarks, tracking adoption and coverage.
• Survey user satisfaction and reproducibility improvements compared to previous evaluation practices (as discussed in Li et al. (2025) and Wang et al. (2025)).

References:

• Li, T., Chen, M., Guo, B., & Shen, Z. (2025). A Survey on Diffusion Language Models. arXiv.org.
• Wang, Y., Yang, L., Li, B., Tian, Y., Shen, K., & Wang, M. (2025). Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models. arXiv.org.