Superposition Meets Intrinsic Dimension: Joint Effects of Data Geometry and Feature Overlap on Scaling Laws

TL;DR: What happens when you mix strong superposition with data that lies on a low-dimensional manifold? For a concrete step, train synthetic and real models on data with controlled intrinsic dimensionality, measuring the interplay between superposition and manifold geometry on scaling exponents.

Research Question: How do the geometric properties of data (e.g., intrinsic dimension) interact with representation superposition to determine neural scaling laws in deep networks?

Hypothesis: The scaling law exponent is not solely determined by model dimension and superposition strength, but also by the intrinsic dimension of the data, as predicted by manifold-based theories (see Havrilla & Liao, 2024).

Experiment Plan: Use synthetic datasets with tunable intrinsic dimension (using manifold learning benchmarks) and real datasets where low-dimensional structure can be estimated. Train models under varying superposition regimes (controlled by weight decay) and measure scaling exponents. Compare empirical exponents to those predicted by Liu et al. (2025) and by low-dimensional theory (Havrilla & Liao, 2024). Analyze interactions: does strong superposition “wash out” data geometry, or do effects combine in nontrivial ways?

References:

• Liu, Y., Liu, Z., & Gore, J. (2025). Superposition Yields Robust Neural Scaling. arXiv.org.
• Havrilla, A., & Liao, W. (2024). Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data. Neural Information Processing Systems.