Who Wins When We Use Less? Group-wise Phase Transitions and Fairness under Curation

TL;DR: Cutting data can raise your average grade but leave some students behind. We develop group-specific scaling/phase diagrams to quantify when aggressive curation improves overall accuracy but harms worst-group performance, and propose fairness-aware curation rules that trade off mean vs min-group risk. Initial experiment: intrusion detection and image-to-anime translation with imbalanced groups to map where “less is more” creates hidden harms.

Research Question: How do less-is-more curation strategies affect worst-group performance and fairness, and can we design curation rules that preserve the benefits without exacerbating disparities?

Hypothesis: When minority groups have higher noise/overlap or are underrepresented, aggressive curation that prioritizes correctness/difficulty can preferentially discard minority examples, improving average accuracy but worsening worst-group metrics. Group-aware objectives can restore worst-group performance while retaining much of the small-data advantage.

Experiment Plan: Setup: Extend Dohmatob’s framework to stratified populations with group-specific label noise and overlap; derive group-wise phase transition curves and worst-group risk bounds under label-aware vs label-agnostic curation.
Data/Tasks:

1. Cyber intrusion detection (Tran et al.; Chen et al.) with known duplication/overlap artifacts—treat different intrusion families as groups; vary deduping/overlap removal.
2. Image-to-cartoon style transfer (Wang et al.) with demographic groups—measure group performance disparities under different curation strengths.
3. Optional: Apply a data-curation rubric (Bhardwaj et al.) to explicitly document fairness-relevant decisions during curation.
   Protocol:
4. Construct curated subsets at increasing “confidence” thresholds (removing duplicates/overlaps vs preserving minority coverage).
5. Train comparable models and measure average vs worst-group accuracy/F1, along with calibration and out-of-group robustness.
6. Introduce a fairness-aware curation criterion that caps per-group discard rates or reweights low-coverage groups and evaluate the resulting scaling curves.
   Analysis: Identify regimes where less-is-more increases disparity; quantify trade-offs and the effectiveness of fairness-aware curation in shifting group-wise phase boundaries.
   Expected Outcome: Evidence that unqualified aggressive curation can harm worst-group performance; practical curation rules that preserve much of the “less is more” benefit while protecting minorities.

References: ['Dohmatob, E., Pezeshki, M., & Askari-Hemmat, R. (2025). Why Less is More (Sometimes): A Theory of Data Curation. arXiv.org.', 'Tran, N., Chen, H., Bhuyan, J., & Ding, J. (2022). Data Curation and Quality Evaluation for Machine Learning-Based Cyber Intrusion Detection. IEEE Access.', 'Chen, H., Tran, N., Thumati, A. S., Bhuyan, J., & Ding, J. (2021). Data Curation and Quality Assurance for Machine Learning-based Cyber Intrusion Detection. arXiv.org.', 'Wang, Y., Cai, Z., & Zhang, Q. (2025). The Effect of Dataset Imbalance on the Performance of Image-to-Cartoon Generative Adversarial Networks. Applied and Computational Engineering.', 'Bhardwaj, E., Gujral, H., Wu, S., Zogheib, C., Maharaj, T., & Becker, C. (2024). Machine learning data practices through a data curation lens: An evaluation framework. Conference on Fairness, Accountability and Transparency.']