Beyond Translation or Transcription: Toward Context-Aware Dynamic Modality Adapters in Spoken Language Models

This research proposes designing a new class of context-aware, dynamic modality adapters (MAs) for spoken language models (SLMs) that can smoothly interpolate between phonetic and semantic representation strategies. Unlike current MAs that are fixed as either phonetic or semantic, these dynamic adapters would adapt based on factors such as the downstream task (recognition, translation, summarization), input language characteristics (e.g., low-resource or unwritten languages, shared writing systems), and model uncertainty (confidence in ASR hypotheses or semantic consistency). Inspired by advances in multimodal adapters in vision-language models, the proposed MAs would use gating mechanisms—such as attention or trainable logistic switches—to fuse or prioritize phonetic and semantic streams dynamically at inference time. Multitask training with explicit losses for phonetic faithfulness and semantic adequacy would encourage robustness and flexibility in both channels. This approach challenges the current design assumption that modality adapters must commit to a single mode and aligns with neurocognitive findings about the brain's flexible integration of multiple representational streams. If successful, this framework could enable SLMs that are more robust in low-resource, code-switching, and cross-lingual scenarios, leveraging the strengths of both semantic understanding and phonetic fidelity without trade-offs.

References:

1. Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Models. Tol'uldop'e Og'unrem'i, Christopher D. Manning, Daniel Jurafsky, Karen Livescu (2025).
2. Multi-Modal Understanding and Generation for Object Tracking. Hong Zhu, Pingping Zhang, Lei Xue, Guangling Yuan (2025). IEEE transactions on circuits and systems for video technology (Print).
3. Modality Confidence Aware Training for Robust End-to-End Spoken Language Understanding. Suyoun Kim, Akshat Shrivastava, Duc Le, Ju Lin, Ozlem Kalinli, M. Seltzer (2023). Interspeech.
4. Pre-training for Spoken Language Understanding with Joint Textual and Phonetic Representation Learning. Qian Chen, Wen Wang, Qinglin Zhang (2021). Interspeech.
5. Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models. Yue Zhang, Hehe Fan, Yi Yang (2024). arXiv.org.
6. A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks. Chia Xin Liang, Pu Tian, Caitlyn Heqi Yin, Yao Yua, An-Hou Wei, Li Ming, Tianyang Wang, Ziqian Bi, Ming Liu (2024). arXiv.org.