We present Mixture of Phonetic Experts based Low-Rank Adaptation (MoPE-LoRA), a parameter-efficient adaptation framework for accented English speech recognition. Recent mixture-of-experts approaches typically allocate experts per accent, implicitly treating accent identity as the primary axis of variation. In contrast, we organize adaptation along phonetic categories, motivated by the observation that accent differences are often expressed as systematic distortions of phoneme realization. MoPE-LoRA introduces a fixed set of six low-rank experts aligned with manner-of-articulation classes and performs frame-level routing through a hybrid mechanism that combines phoneme supervision with learned acoustic gating. Because the expert set is shared across accents, the model does not require accent-specific modules and naturally extends to previously unseen accents. Experiments on L2-ARCTIC show that MoPE-LoRA consistently improves over single LoRA and full fine-tuning. In zero-shot evaluation where one accent is held out during training, the proposed method yields a 12.3% relative improvement over single LoRA. These results indicate that phonetic-category-based decomposition offers a scalable and transferable alternative to accent-level adaptation.
https://arxiv.org/abs/2610.26888
One-shot personalization of autoregressive (AR) text-to-speech (TTS) models remains challenging because supervised fine-tuning (SFT) on a single reference pair rapidly degrades generation quality. We identify fixed, off-policy supervision as a key cause of failure in one-shot SFT for AR TTS, leading to blank or near-empty outputs at inference. Diversifying the training data with zero-shot pseudo-pairs eliminates this failure but yields little improvement in speaker similarity and increases word error rate because the targets remain fixed and teacher-forced. We therefore propose two on-policy approaches for one-shot speaker personalization: Group Relative Policy Optimization (GRPO) with a speaker-similarity reward and On-Policy Self-Distillation (OPSD) with soft supervision from a reference-conditioned teacher. Both on-policy methods improve speaker similarity while preserving intelligibility, with different trade-offs: GRPO achieves the strongest overall performance, whereas OPSD enables substantially faster adaptation without external reward evaluation. Further analysis confirms that these gains stem from on-policy supervision rather than target diversity, establishing on-policy training as an effective framework for one-shot AR TTS personalization.
https://arxiv.org/abs/2610.17838
End-to-end speech-to-text retrieval is becoming an important capability for voice agents, allowing spoken queries to retrieve relevant documents without intermediate transcription. Training these retrievers typically depends on large collections of spoken query-document pairs that are costly to acquire and difficult to extend across domains and languages. We introduce \ours, which replaces such task-specific supervision with speech-transcript pairs from standard ASR corpora. A lightweight adapter projects the output of a frozen speech encoder into the retrieval space of a frozen pretrained text embedding model. \ours is competitive with cascaded ASR-then-retrieve systems while reducing inference latency, and outperforms existing publicly available speech embedding models. It also transfers to text-to-speech and speech-to-speech retrieval without additional training. Controlled analyses validate the speech-transcript alignment paradigm, demonstrating robustness to reduced training data and consistent gains from larger corpora. Further ablations of loss functions and trainable modules establish practical guidelines for the framework. Model weights and codes are to be released.
https://arxiv.org/abs/2610.17813
Electrolaryngeal (EL) speech suffers from severe acoustic mismatches with normal speech, resulting in degraded naturalness and intelligibility. These mismatches also make sequence-to-sequence voice conversion struggle to learn robust intermediate representations, hindering EL-speech-to-normal-speech conversion (EL2SP) performance. To address this, we propose a novel multimodal contrastive representation learning framework to 1) achieve richer multimodal-representation learning, and 2) facilitate the transferability of multimodal representations to speech-only EL2SP. First, EL speech and text representations are fused to learn multimodal representations for target speech generation, during which cross-modal contrastive learning is employed to encourage a semantically consistent speech--text representation space. Next, the fused representations are treated as a fixed target to guide speech-only representation learning through cross-stage contrastive learning, enabling multimodal knowledge to be transferred to the EL2SP task while preserving architectural and inference simplicity. Experimental results demonstrate that the proposed method consistently outperforms both the speech-only and multimodal-based baselines.
https://arxiv.org/abs/2610.17716
Real-world audio recordings for deepfake detection rarely consist solely of speech; they often are mixtures of speech and background events that themselves may also be synthetic. When processing these mixtures, it remains unclear whether detectors attend to the speech or the background event, and where their spectral evidence lies. We study this on the CompSpoof V2 database, whose mixtures pair independently real-or-fake speech and event components. Our results show that all systems detect fake events more accurately than fake speech. However, this performance gap is largely driven by poor generalization to an unseen fake speech domain, rather than a true difference in difficulty. Furthermore, sub-band probing reveals that the front-end encoder dictates where a countermeasure focuses its frequency evidence. To enable reliable deployment in real-world scenarios, future countermeasures should address out-of-domain generalization failures and carefully select front-end encoders aligned with their target acoustic evidence. Our source code and checkpoints are released at this https URL.
https://arxiv.org/abs/2610.17617
Dynamic spatial audio reasoning requires a model to associate changing acoustic observations with the correct source and recover its spatial state at a queried time or interval, a challenge intensified by overlapping speech and source motion. To study this problem, we introduce Sonic-Bench, comprising 2.8K first-order Ambisonics clips and 28.8K target-specific question-answer pairs across 480 simulated indoor scenes. The benchmark covers localization, distance estimation, and motion reasoning in single- and dual-source settings. We further propose SonicWorld, which integrates physical, learned acoustic, and semantic features into persistent source states through learned transitions and recurrent updates, and projects them into an Omni LLM for spatial reasoning. Compared with Spatial-Omni on the Sonic-Bench test split, SonicWorld improves direction accuracy from 31.98% to 34.69% and reduces azimuth error from 57.22° to 48.01°. In zero-shot transfer to Spatial LibriSpeech, SonicWorld retains strong localization performance, achieving 73.98% direction accuracy and 19.29° azimuth error, compared with 27.14% and 70.74° for Spatial-Omni.
https://arxiv.org/abs/2610.17131
Large audio-language models (LALMs) have become highly capable on a wide range of tasks spanning speech, music, and environmental sound. However, many modern LALMs resample input audio to 16 kHz mono, making all frequencies above 8 kHz unobservable. We present SpectralBridge, a lightweight extension that gives such a model access to the 8-16 kHz band by joining low- and upper-band representations with time-aligned paired cross-frequency attention. Applied to Audio Flamingo Next, the method trains only a small adapter and rank-8 LoRA modules, while leaving the original audio encoder, audio projector, and base language model frozen. SpectralBridge improves both zero-shot transfer and downstream fine-tuning: it raises zero-shot balanced accuracy on the ESC-50 audio classification dataset from 75.5 to 90.5, and when fine-tuned on Insect-10, a 10-way insect species task we construct from InsectSet459 recordings, under the same budget as a LoRA fine-tuned baseline, it reaches 55.4 balanced accuracy against the baseline's 26.2. On a benchmark where the answer depends only on the 8-16 kHz band, it gains 30 mAP over the frozen model. Erasing the upper-band stream from the trained model removes these gains, showing that they come from the high-frequency signal itself rather than from extra tokens or the adapted prompt format.
https://arxiv.org/abs/2610.17048
Whisper-style ASR models provide strong zero-shot transcription, but their residual errors are not evenly distributed across datasets, recording conditions, and metadata-based groups. We study when lightweight low-rank adaptation (LoRA) improves Whisper and whether observed gains require group-specific adapters. Across Common Voice, LibriSpeech, and VoxPopuli, we compare zero-shot Whisper, pooled LoRA, and group-specific LoRA under matched within-group splits. We find that adaptation is not uniformly beneficial: gains are concentrated in difficult Common Voice groups, are consistent on VoxPopuli, and largely disappear on low-error LibriSpeech. Crucially, pooled LoRA often matches, and sometimes clearly outperforms, group-specific LoRA. Gains over zero-shot should therefore be tested against matched pooled adaptation before being interpreted as evidence of personalization.
https://arxiv.org/abs/2610.17019
Standard hate speech detection datasets often emphasize explicit harmful language, but directed harm can also be expressed indirectly, without clearly hateful wording. Sarcasm is one such form of indirect expression, as it can convey harmful intent through mock praise, exaggeration, humor, or apparent politeness. In this paper, we introduce SarcasTraps, a targeted external test set for analyzing the effect of sarcasm on Turkish hate speech detection. SarcasTraps contains 479 entries annotated along two dimensions: hate speech and sarcasm. We train classifiers using BERTurk and XLM-RoBERTa on two existing Turkish hate speech datasets and evaluate them on SarcasTraps, using it strictly as an external test set. We also evaluate GPT-4.1 and Llama-3.3-70B in a zero-shot setting on SarcasTraps. Results show that sarcasm substantially affects hate speech detection. Fine-tuned encoder models lose recall on sarcastic hate, while zero-shot LLMs maintain higher recall but suffer substantial precision drops on sarcastic examples. These findings show that both fine-tuned hate speech detectors and zero-shot LLMs struggle when sarcasm is involved, and SarcasTraps provides quantitative evidence of this challenge through a targeted Turkish test set. Due to the presence of sensitive and hateful language, SarcasTraps will be shared upon request for academic research purposes. Warning: This paper has examples from the dataset which contains offensive and hateful content towards groups of people.
https://arxiv.org/abs/2610.16957
Target speaker extraction (TSE) aims to recover a target speaker's speech from a speech mixture using an enrollment utterance. Recent codec-based generative approaches formulate TSE as conditional speech token generation and achieve promising perceptual quality. However, most existing systems rely on unified codec tokens that entangle linguistic content, speaker characteristics, and acoustic details, forcing a single model to jointly handle target selection, content semantic, and acoustic reconstruction. In this paper, we investigate whether disentangled semantic-acoustic codec representations are better suited to TSE. We first show that simply replacing unified codec tokens with semantic-acoustic tokens is insufficient and can even degrade performance when both remain handled by a unified generative model. Motivated by this observation, we propose a task-aware discriminative-generative framework, where a discriminative branch extracts target-related semantic information, while a generative branch predicts coarse acoustic tokens and refines them under semantic guidance. This specialization separates what is said from how it sounds while coordinating both factors in a factorized codec space. Experiments on Libri2Mix show that our method reduces dWER from 0.388 to 0.118 compared with a unified-codec baseline, while maintaining competitive perceptual quality and speaker consistency. Ablation studies confirm that the gains arise from task-aware semantic-acoustic modeling rather than representation disentanglement alone.
https://arxiv.org/abs/2610.16858
Deepfake speech detectors are typically evaluated as binary classifiers on fixed benchmarks, but deployed systems face evolving attacks, codecs, speakers, and recording conditions, with labels unavailable during operation. This makes it difficult to know when a detector operates in familiar conditions and can be relied on. We propose an unsupervised drift monitor that operates on the internal embeddings of a deployed detector. The monitor compares each input against bona fide and spoof references using regularized Mahalanobis distances. These distances define trusted, ambiguous, and novel regions of the detector embedding space, allowing unfamiliar inputs to be routed for review. We evaluate the monitor on four modern detectors on three datasets. The monitor raises alerts for dataset mismatches and substantially reduces the security-critical case of accepting deepfakes at the cost of reduced automatic coverage. The results also show that drift affects both classes: bona fide recordings can move away from the training reference, increasing alert rates even in in-domain settings. Our findings suggest that deployed deepfake speech detectors should be treated not only as a binary classification, but also as a reliability problem.
https://arxiv.org/abs/2610.16826
Code-switching (CS) speech, characterized by language alternation within a single utterance, is prevalent in multilingual communities and poses challenges to automatic speech recognition (ASR) due to rapid language transitions. We propose LLM-HB, a language-aware large language model-guided hotword biasing framework for code-switching ASR (CS-ASR). The framework incorporates a mixture-of-experts (MoE) adaptor and an auxiliary language prediction head to better capture and differentiate multilingual representations, while hotwords are injected through LLM-based prompting. Experiments on the ASRU2019 code-switching challenge test set demonstrate that LLM-HB achieves a 20.30% relative reduction in mixed error rate (MER) over the baseline, reaching 5.85% MER under a 15-distractor hotword setting. To the best of our knowledge, this is the first work to integrate LLM-guided biasing into code-switching ASR. We further release the extracted ASRU2019 hotword list to support future research.
https://arxiv.org/abs/2610.16579
Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated speech and sound often temporally misalign with visual events. A key bottleneck is the lack of human-annotated data, reliable evaluation benchmarks, and synchronization-aware evaluation models for generated audio--visual outputs. To address these limitations, we conduct a comprehensive study spanning data, benchmarking, and reward modeling. We first introduce SyncReward-Data-80K, a dataset containing 80K professional synchronization ratings for clips from eight audio--visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. We further construct \textbf{SyncReward-Bench}, a held-out benchmark of 1K clips, each independently rated by multiple professionals. Using this benchmark, we systematically evaluate specialized synchronization metrics and omni-modal models, revealing their substantial misalignment with human judgments. Finally, we develop SyncReward, a reward model that combines semantic--temporal contrastive pretraining on real clips with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of-$N$ selection and reinforcement learning based post-training. We will release the data, benchmark, model, and code at this https URL.
https://arxiv.org/abs/2610.16338
Continuous autoregressive speech language models combine an autoregressive (AR) backbone with a diffusion or flow-matching head. Preference post-training requires choosing how to construct pairs and which modules to update. Controlled interventions show that a perturbed acoustic patch changes subsequent backbone states, yet larger representation changes do not necessarily reduce speech quality. We construct preference pairs through noise or inference-configuration perturbations and formulate a preference update based on reference-relative flow-matching errors. Updating the backbone with a fixed head yields stronger reward improvements than head-only adaptation in our evaluated setting. Across flow-time and module ablations, the lowest training loss does not consistently produce the best speech. These findings support preference post-training that considers both acoustic sampling and autoregressive conditioning, with training choices validated on generated utterances.
https://arxiv.org/abs/2610.16276
Current ASR models perform well on general benchmarks but still struggle with acoustically ambiguous words. Visual context can aid recognition in screen-based scenarios, yet existing benchmarks provide limited insight into the benefits of different types of visual evidence. To address this gap, we introduce ScreenASR-Bench , a tiered benchmark with three progressively challenging levels: Explicit Textual Evidence (ETE), based on direct on-screen text; Explicit Visual Evidence (EVE), using visible content rather than on-screen text; and Implicit Semantic Evidence (ISE), requiring scene-level inference. Extensive experiments on ASR models, Large Audio-Language Models (LALMs) and Omni models show that incorporating visual context substantially improves recall over the audio-only setting across all three model families. The gains are largest for ETE and smallest for ISE, indicating that models leverage direct visual evidence more effectively than indirect semantic cues. To facilitate future research on visually grounded speech recognition, we publicly release ScreenASR-Bench.
https://arxiv.org/abs/2610.16260
Spatially encoded speech conveys not only linguistic information but also spatial cues. This enables listeners to distinguish and localize speakers, contributing to more natural and realistic auditory experiences in applications such as remote conferencing and immersive media. Neural speech codecs (NSCs) enable high-quality reconstruction of monaural speech at increasingly low bitrates while preserving linguistic information, but their ability to effectively preserve spatial cues remains limited. We propose FOAdapter, a plug-and-play spatial tokenizer that leverages a pretrained monaural speech codec to efficiently represent both linguistic and spatial information in first-order ambisonic (FOA) speech at low bitrates. Comprehensive evaluation demonstrates that FOAdapter consistently preserves spatial information across all evaluated codecs. Source code and audio examples are available on our demo website.
https://arxiv.org/abs/2610.16202
Nonverbal vocalizations (NVVs), such as laughter and sighs, convey meaning alongside words. NVV-aware speech systems must recognize and generate these events while preserving lexical accuracy and perceptual quality. Yet differences in annotation, resources, and evaluation complicate comparisons, and aggregate scores can obscure specific weaknesses. We review ten system descriptions from the NVVSpeech Challenge 2026, covering understanding and generation in Chinese and English across 16 event categories. Our analysis examines supervision construction, task adaptation, and inference-time or preference-based feedback. Results reveal persistent recognition difficulties with breath and gasp and trade-offs between event recognition and lexical accuracy, as well as between generation intelligibility and acoustic quality, with human listening providing complementary evaluation evidence. These findings identify priorities for advancing NVV-aware speech understanding, controllable generation, and comparable evaluation.
https://arxiv.org/abs/2610.16076
The Flow Matching objective supervises the velocity field at intermediate states, whereas waveform metrics are evaluated after decoding and may be non-differentiable. To align Flow Matching models with waveform-level and perceptual quality objectives during post-training, FlowGRPO optimizes their trajectories using endpoint rewards. In this work, we focus on music separation and use multi-resolution short-time Fourier transform (MRSTFT), scale-invariant signal-to-distortion ratio (SI-SDR), and Virtual Speech Quality Objective Listener (ViSQOL) as reward components. Before policy optimization, we analyze the relationship between data quality factors and the within-group gap between the maximum and mean rollout rewards for each component. This analysis motivates a revised ViSQOL reward that matches channel-wise root-mean-square (RMS) levels and evaluates the interval from the first to the last active frame, retaining inactive frames within that interval. We also apply a sample-quality filter based on predefined waveform checks. Together, the revised reward and quality filter enable FlowGRPO to improve the pretrained vocal separator in both separation performance and perceptual quality.
https://arxiv.org/abs/2610.15780
Eyewear that senses articulation with inaudible ultrasound can now recognize silent speech over an open vocabulary, but its accuracy is limited by the cost of training data. We present SoniGAN, which synthesizes the ultrasonic echo profiles of silent articulation from text and uses them as synthetic supervision. A voice-cloned text-to-speech (TTS) model reads a text corpus, and a two-stage generator synthesizes echo profiles from representations of the spoken content of that audio. Pretraining a recognizer on 104 hours of the synthesized data improves the word error rate (WER) of a public open-vocabulary benchmark by 2.3 points. Adding the sessions the corpus discards for a faulty microphone brings the system to 22.2% without any language model. Lexicon-constrained decoding, LLM rescoring, and ensembling further bring it to 14.9%, the first minimally obtrusive silent speech interface to fall inside the error range in which users report no loss of willingness to use silent speech input.
https://arxiv.org/abs/2610.15671
We present AnyEdit, a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments, including noise, reverberation, and background music. Recent unified generative models already achieve strong speech and singing generation in clean conditions, but real-world vocal editing remains challenging because the model must revise content while simultaneously preserving speaker identity, prosody, and the surrounding acoustic scene. Directly modeling noisy audio end-to-end entangles semantic editing with environmental acoustics, weakening the core vocal generation capability. We therefore adopt a two-stage design that preserves clean-domain editing ability while deferring acoustic rendering to a separate stage: an instruction-guided autoregressive editor first predicts clean prosody and content-style tokens, supported by a teacher-student noise-robust prosody tokenizer that distills clean prosodic tokens from degraded recordings; a flow-matching acoustic model then performs in-context rendering conditioned on the source acoustic context, so the edited segment inherits surrounding noise, reverberation, and background sound while maintaining consistent speaker timbre. To stabilize unedited regions, we introduce unmodified-area-aware supervision; after pre-training, we further apply direct preference optimization on mined real-to-real singing pairs. We also release AnyEditBench, a benchmark covering both speech and singing voice editing under clean, simulated, and in-the-wild acoustic conditions. Experiments on Chinese and English speech and singing datasets showed that AnyEdit consistently outperforms strong baselines in content accuracy, subjective quality, and background consistency under noisy and reverberant conditions. Demo: this https URL. AnyEditBench: this https URL.
https://arxiv.org/abs/2610.15665