Adaptive Training Schedules for Expanding the Generalization Window in Diffusion Models

TL;DR: What if we could dynamically adjust training schedules so that diffusion models spend more time in the “safe” generalization window and less time near memorization? Let’s try curriculum-style or dataset-size-aware training rate adjustments and see if it extends the window before memorization kicks in.

Research Question: Can adaptive, data-size-aware training schedules prolong the generalization phase in diffusion models, thus mitigating the onset of memorization even for moderate dataset sizes?

Hypothesis: By adjusting learning rates or introducing scheduled early-stopping signals based on real-time monitoring of $\tau_\mathrm{gen}$ and $\tau_\mathrm{mem}$, we can extend the period where generalization dominates, effectively raising the threshold dataset size required for memorization to emerge.

Experiment Plan: - Train diffusion models (e.g., U-Nets) on datasets of varying sizes.

• Implement dynamic schedulers that monitor training metrics (e.g., sample diversity, overfitting signals) to slow or halt training as the model approaches $\tau_\mathrm{mem}$.
• Compare generalization and memorization metrics (using, e.g., FID, overfitting diagnostics from Wen et al., 2024) to standard fixed training schedules.
• Assess if adaptive schedules increase the effective window between $\tau_\mathrm{gen}$ and $\tau_\mathrm{mem}$.

References:

• Bonnaire, T., Urfin, R., Biroli, G., & M'ezard, M. (2025). Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training. arXiv.org.
• Wen, Y., Liu, Y., Chen, C., & Lyu, L. (2024). Detecting, Explaining, and Mitigating Memorization in Diffusion Models. International Conference on Learning Representations.