Human text is no longer useful in a post-scaling world

Idea: The "adaptation" procedure creates divergence from human language. Moving forward, human language will no longer be needed for building models.

More details:

As we train models for tasks requiring specific capabilities (e.g., SWE), they might become worse at modeling the natural distribution of language (e.g., C4) or even language of specific tasks (e.g., Minerva human-written answers) while prefer their own model-generated sets.

This is interesting because (1) it contradicts our opinions of scaling on human data and (2) it may mean we have a future of model training without human text involved

Somewhat related work on LLM response similarity (to each other): https://arxiv.org/abs/2502.16173