Beyond Compression: Detecting and Explaining Scaling Law Anomalies in Hierarchical Language Models

TL;DR: Let’s find out where and why DLCM’s scaling laws break! By systematically probing benchmarks and compression regimes, we can uncover when the model’s performance or efficiency unexpectedly deviates and try to explain these anomalies. For instance, we can hypothesize that certain linguistic genres or tasks—like poetry or multi-hop reasoning—will violate predicted trade-offs between compression and reasoning, offering new insights into model limits.

Research Question: Where do DLCM’s compression-aware scaling laws fail to predict actual efficiency or reasoning performance, and what linguistic or architectural factors explain these deviations?

Hypothesis: Some semantic domains or reasoning tasks will exhibit scaling law “anomalies”—unexpected drops or gains in efficiency or accuracy—due to factors like ill-defined concept boundaries, context fragmentation, or heterogeneity in information density that hierarchical compression cannot capture.

Experiment Plan: Compile a diverse set of language tasks and benchmarks, focusing on those with known high semantic complexity (multi-hop QA, figurative language, long-range dependencies). Systematically vary the compression ratio (R) and measure deviations from DLCM’s predicted scaling curves for reasoning accuracy and compute efficiency. Use error attribution (e.g., from ConvBench: Liu et al., 2024) to link anomalies to specific concept transitions or failure points. Analyze the linguistic features or input structures associated with anomalous behavior, possibly using concept boundary visualizations or diagnostic probes.

References:

• Qu, X., Wang, S., Huang, Z., Hua, K., Yin, F., Zhu, R.-J., Zhou, J., Min, Q., Wang, Z., Li, Y., Zhang, T., Xing, H., Zhang, Z., Song, Y., Zheng, T., Zeng, Z., Lin, C., Zhang, G., & Huang, W. (2025). Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space.
• Liu, S., Ying, K., Zhang, H., Yang, Y., Lin, Y., Zhang, T., Li, C., Qiao, Y., Luo, P., Shao, W., & Zhang, K. (2024). ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Models. Neural Information Processing Systems.