Compute-aware Less-is-More: Phase Diagrams Coupling Curation, Repetition, and Budget

TL;DR: Like packing for a trip, the best suitcase depends on how long you travel—picking only the best items works for a weekend, but you want more variety for a month. We propose a theory and empirical study of how the benefit of aggressive curation depends on compute budget and the inevitable repetition it induces. Initial experiment: for fixed compute, vary the fraction of high-quality curated data and the number of epochs; fit a 3D “phase diagram” of test error E(N, Q, C) to predict when small curated sets outperform large, noisier ones.

Research Question: How do training compute budget and data repetition interact with data curation to create budget-dependent “less-is-more” regimes?

Hypothesis: There exists a compute-dependent crossover: at low-to-moderate budgets, aggressively curated small datasets outperform larger, noisier datasets; as compute increases and curated data must be repeated more often, the marginal utility of curation diminishes, and adding lower-quality but unseen data becomes optimal. This crossover is governed by a repetition penalty that shifts the phase boundaries derived by Dohmatob et al.

Experiment Plan: Setup: Extend the imperfect-oracle model from Dohmatob et al. by adding (i) a repetition-dependent utility decay as in Goyal et al., and (ii) a compute budget C that determines (epochs × batch size × steps). Derive a predicted performance surface E(N, Q, C) and identify phase boundaries.
Data/Tasks: Large-scale vision-language pretraining (DataComp/LAION-like pools), plus smaller-scale classification (ImageNet-1k). Build multiple data “buckets” stratified by estimated quality (DFN-style filtering).
Protocol:

1. Train CLIP-like models at multiple fixed compute budgets (e.g., 25%, 50%, 100% of a baseline) using: (a) curated-only with varying repetition; (b) mixed curated + uncurated to avoid repetition; (c) unfiltered baselines.
2. Repeat across different curation strengths and DFN variants.
3. Measure zero-shot accuracy, linear-probe accuracy, and retrieval on standard suites; fit the scaling surface E(N, Q, C).
   Analysis: Quantify the repetition penalty and identify compute thresholds where curated datasets flip from superior to inferior. Validate theory by matching predicted versus observed phase boundaries.
   Expected Outcome: Curated subsets should dominate at low compute; their advantage shrinks and reverses as repetition increases with compute, confirming that curation is not compute agnostic.

References: ['Dohmatob, E., Pezeshki, M., & Askari-Hemmat, R. (2025). Why Less is More (Sometimes): A Theory of Data Curation. arXiv.org.', 'Goyal, S., Maini, P., Lipton, Z. C., Raghunathan, A., & Kolter, J. (2024). Scaling Laws for Data Filtering—Data Curation Cannot be Compute Agnostic. Computer Vision and Pattern Recognition.', 'Fang, A., Madappally Jose, A., Jain, A., Schmidt, L., Toshev, A., & Shankar, V. (2023). Data Filtering Networks. International Conference on Learning Representations.', 'Lin, F., Hu, Y., Sheng, P., Wen, C., You, J., & Gao, Y. (2024). Data Scaling Laws in Imitation Learning for Robotic Manipulation. International Conference on Learning Representations.']