Ultra-Fast Sampling via Adaptive Score Network Pruning in Diffusion Models

Li & Cai (2024) theoretically accelerate diffusion sampling, but most practical diffusion models remain slow due to dense score network evaluations. This idea proposes an adaptive pruning mechanism: during sampling, the score network is analyzed for redundant or low-importance components (e.g., neurons, attention heads, or even whole layers), which are temporarily pruned or sparsified for that particular sample or sampling phase. The model could learn a policy for pruning that trades off speed and fidelity, possibly conditioned on the difficulty of the sample being generated. This would allow for “on-demand” acceleration in settings where latency is critical (e.g., real-time 3D scene generation from Gao et al., 2024, or live recommendation), and could be theoretically grounded with convergence guarantees similar to those in Li & Cai (2024). No current work combines pruning/sparsity with provable fast sampling in diffusion models, making this a fresh direction.

References:

1. CAT3D: Create Anything in 3D with Multi-View Diffusion Models. Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, Ben Poole (2024). Neural Information Processing Systems.
2. Provable acceleration for diffusion models under minimal assumptions. Gen Li, Changxiao Cai (2024). arXiv.org.