A Meta-Evaluation Framework for Spoken Language Models: Unified Cross-Task Benchmarking and Taxonomy Using Emergent Model Behaviors

Building on the comprehensive landscape outlined in "On the landscape of spoken language models," this research addresses the fragmentation in task-specific evaluations and the lack of meta-level understanding of spoken language model (SLM) behaviors across tasks. It proposes a meta-evaluation framework that aggregates cross-task performance data while tracking and analyzing unexpected model behaviors such as coherence failures, catastrophic forgetting, and outlier performances on new modalities or under-represented languages. The framework introduces a dynamic taxonomy of SLM capabilities based on observed outcomes, new meta-metrics that capture variance, surprise, and robustness, and meta-analysis tools for automated discovery and visualization of outlier patterns and correlations. It also incorporates evaluation by large language models (LLMs) and human-in-the-loop processes for ambiguous or novel failure cases. This approach focuses on deviations and emergent behaviors rather than average performance, turning outliers into actionable insights for model improvement, new task design, and identifying domains where SLMs develop unexpected strengths. Long-term goals include accelerating identification of generalizable or failing architectures, informing next-generation pretraining strategies, and fostering unified, community-driven benchmarks prioritizing robustness and coverage over isolated task performance.

References:

1. On The Landscape of Spoken Language Models: A Comprehensive Survey. Siddhant Arora, Kai-Wei Chang, Chung-Ming Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung-yi Lee, Karen Livescu, Shinji Watanabe (2025). arXiv.org.