Alignment-First Curation: Predicting When Less Pretraining Beats More Under Distribution Shift

TL;DR: If you’re studying for a French exam, fewer high-quality French books beat lots of Spanish ones. We formalize an “alignment-aware” curation rule that prioritizes distributional match to the downstream task and test where it inverts classical “more is better” scaling. Initial experiment: in machine translation, vary both size and alignment of pretraining corpora and map when small aligned subsets outperform larger misaligned pools; repeat in scientific domains using synthetic-but-aligned data.

Research Question: When does prioritizing distributional alignment via curation outperform increasing upstream data size for downstream performance?

Hypothesis: With sufficient pretrain–downstream alignment, smaller curated datasets can outperform larger misaligned datasets on downstream metrics; the alignment term shifts Dohmatob et al.’s phase transitions and explains non-monotonic downstream curves observed in transfer settings.

Experiment Plan: Setup: Extend the theory to include an alignment parameter A that modulates the effective data quality. Predict regimes where increasing pretraining size with poor alignment degrades downstream BLEU/COMET even as upstream loss improves.
Data/Tasks:

1. Machine Translation (following Isik et al.): vary the pretraining corpus alignment to downstream language pairs/domains; finetune LLMs and evaluate BLEU/COMET.
2. Solar flare forecasting (Newman et al.): curate SHARP parameters by excluding multi-AR patches and prioritizing flaring-history-aligned samples; evaluate lead-time-dependent forecasting accuracy.
3. Electron microscopy (Eliasson & Erni): generate synthetic STEM images using the fast multislice surrogate to produce highly aligned data to the downstream size-estimation task; compare to larger, heterogeneous real-only sets.
   Protocol: Construct families of datasets that trade off size vs alignment. Train identical models across these families to obtain downstream performance curves and estimate alignment-aware scaling laws.
   Analysis: Fit log-laws incorporating A; quantify crossover points where small aligned subsets dominate. Test whether alignment reduces variance and shifts phase boundaries in predictable ways.
   Expected Outcome: Clear regimes where alignment-first curation wins; synthetic aligned data can substitute and amplify alignment when real aligned data is scarce.

References: ['Dohmatob, E., Pezeshki, M., & Askari-Hemmat, R. (2025). Why Less is More (Sometimes): A Theory of Data Curation. arXiv.org.', 'Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vassilvitskii, S., & Koyejo, S. (2024). Scaling Laws for Downstream Task Performance in Machine Translation. International Conference on Learning Representations.', 'Newman, T. S., Hall, C. W., Farris, L., Singh, T., Pogorelov, N. V., Benson, B., Raza, S. A. Z., & Trital, P. (2025). Solar Flare Forecasting Using Machine Learning and SDO/HMI Data: A Multiple Machine Learning Model and Data Curation Technique Comparison Study. Astrophysical Journal Supplement Series.', 'Eliasson, H., & Erni, R. (2025). Improving Nanoparticle Size Estimation from Scanning Transmission Electron Micrographs with a Multislice Surrogate Model. Nano Letters.', 'Xerxa, E., Vogt, M., & Bajorath, J. (2024). Influence of Data Curation and Confidence Levels on Compound Predictions Using Machine Learning Models. Journal of Chemical Information and Modeling.']