Exponent-Aware Curation: Selecting Examples by Their Individual Scaling Behavior

TL;DR: Some training examples are great teachers early on; others only help once you’re advanced—pick the early teachers when data is scarce. We propose estimating per-example scaling exponents and curating subsets to maximize gains at a target data budget. Initial experiment: learn Covert-style individualized scaling behaviors with a small number of probes, then select points with the steepest small-N payoff.

Research Question: Can per-example scaling exponents guide curation to outperform difficulty- or correctness-based selection at given data budgets?

Hypothesis: Subsets that maximize the aggregate small-N exponent (weighted by estimated correctness) outperform heuristic curation for a fixed N and compute. As N grows, the optimal subset smoothly transitions to include examples with large-data exponents, producing predictable “curation crossover” behavior.

Experiment Plan: Setup: Use the amortized estimator from Covert et al. to infer each point’s scaling exponent from a handful of training runs. Define a budget-aware objective that selects a subset maximizing expected improvement at the target N.
Data/Tasks:

1. ImageNet-1k classification.
2. Chemistry classification with sequential curation (Xerxa et al.) and a second dataset (aromatase inhibitors; Abdelkrim et al.).
   Protocol:
3. Collect small pilot runs at multiple subset sizes to estimate per-point exponents.
4. Compare four selection strategies: exponent-aware, Dohmatob-style difficulty/correctness oracle, DFN, and random.
5. Train final models at target budgets and evaluate test accuracy/AUROC; repeat across budgets to observe crossover.
   Analysis: Quantify gains in small-N regimes; assess stability of exponent estimates and their transferability across model classes. Check whether exponent-aware subsets better separate classes in feature space (as observed by Xerxa et al. for curated chemistry sets).
   Expected Outcome: Exponent-aware curation yields the strongest small-budget performance and predicts where LIMO-like benefits taper off as N increases.

References: ['Dohmatob, E., Pezeshki, M., & Askari-Hemmat, R. (2025). Why Less is More (Sometimes): A Theory of Data Curation. arXiv.org.', 'Covert, I., Ji, W., Hashimoto, T. B., & Zou, J. (2024). Scaling Laws for the Value of Individual Data Points in Machine Learning. International Conference on Machine Learning.', 'Fang, A., Madappally Jose, A., Jain, A., Schmidt, L., Toshev, A., & Shankar, V. (2023). Data Filtering Networks. International Conference on Learning Representations.', 'Xerxa, E., Vogt, M., & Bajorath, J. (2024). Influence of Data Curation and Confidence Levels on Compound Predictions Using Machine Learning Models. Journal of Chemical Information and Modeling.', "Abdelkrim, A., Bouramoul, A., Zenbout, I., & Brahimi, S. (2023). Evaluating the Effectiveness of Machine Learning Models for Classifying Chemical Inhibitors: A Case Study of Aromatase Inhibitors and PubChem's Molecular Fingerprint Descriptors. 2023 International Conference on Networking and Advanced Systems (ICNAS)."]