Semantic-Acoustic Coevolution: Dynamic Mutual Feedback for Enhanced Spoken Language Modeling

Flow-SLM (Chou et al., 2025) models semantic (discrete) and acoustic (continuous) features jointly using flow-matching to predict acoustics conditioned on content, but it does so asymmetrically and unidirectionally. This research proposes a coevolutionary architecture where semantic tokens and acoustic representations are iteratively updated in a bidirectional loop. At each iteration, the model predicts acoustic vectors from semantic tokens, refines semantic tokens based on the generated acoustics and context, and repeats until consistency is reached. This dynamic mutual feedback aims to better capture nuanced co-articulation, emotional intonation, and prosodic emphasis, enable controllable generation based on emotion or content, and reveal bidirectional dependencies in spoken language. The approach contrasts with static or sequential models by making semantic and acoustic layers mutually constitutive, potentially leading to more natural, context-sensitive speech generation and stronger inductive biases for unsupervised or low supervision learning.

References:

1. Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modeling. Ju-Chieh Chou, Jiawei Zhou, Karen Livescu (2025). arXiv.org.