Breaking the Superposition Barrier: Empirical Identification and Manipulation of Scaling Law Deviations in LLMs

TL;DR: Let’s poke at where and why neural scaling laws break down by intentionally perturbing the degree of superposition in real LLMs, and see if we can predict or induce anomalies. For a first experiment, systematically vary weight decay and data distributions in open-source LLMs, looking for sharp or unexpected changes in scaling behavior (loss vs. model size).

Research Question: Under what conditions does the inverse-dimension scaling law predicted by strong superposition break down in practical LLMs, and can we anticipate or control these deviations?

Hypothesis: There exist “superposition transition points” in model size, weight decay, or data feature distributions where the loss scaling law shifts or fails, and these transitions can be mapped and potentially predicted by geometric or statistical diagnostics.

Experiment Plan: Take a suite of open-source LLMs (e.g., GPT-2, LLaMA variants), train them with varying weight decay, and on datasets with manipulated feature frequency distributions (from power-law to uniform). Measure model loss as a function of model size and compare to the predicted scaling regimes (power law, inverse dimension, or breakdown). Identify and characterize points where empirical scaling deviates from theoretical predictions; analyze representation geometry and overlap. If anomalies arise, test if these correspond to qualitative changes in feature superposition (e.g., via probing or feature disentanglement metrics).

References:

• Liu, Y., Liu, Z., & Gore, J. (2025). Superposition Yields Robust Neural Scaling. arXiv.org.