Semantic Disentanglement Maps: Visualizing LLM Benchmark Relationships Beyond Perplexity

Building on the approach of "Mapping Overlaps in Benchmarks through Perplexity in the Wild," which adeptly uses perplexity patterns and salient token signatures to characterize and relate benchmarks, this idea pushes further into the "why" of benchmark overlap and divergence by bringing in ensemble semantic and neural representations. While the original work highlights that performance overlaps between benchmarks are widespread—even when semantic content is not strongly shared—current analyses often remain at the lexical or token level (perplexity) and don’t deeply connect semantic or representational layers with performance quirks. Papers like Mustafazade & Ebbinghaus (2022) and Haider Ali et al. (2025) demonstrate that semantic similarity metrics (e.g., BERTScore, CLIP-Score) can provide fine-grained, contextual alignment information, yet such metrics are rarely synthesized with perplexity-based approaches or mapped in a multidimensional space. The idea is to integrate three axes for each benchmark: 1) Perplexity signature as defined in Wu et al. (2025)—capacity familiarity signatures; 2) Semantic similarity landscape calculated using state-of-the-art semantic similarity and answer evaluation frameworks; 3) Model-specific activation or attention trace signatures using neural interpretability techniques. By constructing "Semantic Disentanglement Maps," the research aims to visualize how perplexity-driven capacity overlaps do and do not mirror semantic similarity and neural representation similarities across benchmarks. This synthesis allows researchers to identify benchmarks with high performance overlap but divergent semantic and representation layers, highlight outlier or orthogonal benchmarks, and provide actionable insights for benchmark designers to diagnose or mitigate misleading correlations in model assessment. Unlike current literature, this work moves from tabular correlations and linear regressions to rich, explainable visualizations and a triaxial analysis, uniquely combining perplexity, semantic evaluation metrics, and neural representations—a synthesis not yet explored. The impact is both methodological, offering a practical toolkit for benchmark designers and LLM developers, and scientific, challenging assumptions about benchmark performance overlap and suggesting new directions for finer-grained LLM evaluation and model development. In short, Semantic Disentanglement Maps would offer the first truly holistic "landscape view" of LLM evaluation—bridging gaps between statistical exposure, semantic depth, and neural processing.

References:

1. Evaluation of Semantic Answer Similarity Metrics. Farida Mustafazade, Peter F. Ebbinghaus (2022). Machine Learning & Applications.
2. A Semantic Evaluation Framework for Medical Report Generation Using Large Language Models. Haider Ali, Rashadul Islam Sumon, Abdul Rehman Khalid, Kounen Fathima, Hee Cheol Kim (2025). Computers, Materials & Continua.
3. Mapping Overlaps in Benchmarks through Perplexity in the Wild. Siyang Wu, Honglin Bao, Sida Li, Ari Holtzman, James A. Evans (2025). arXiv.org.