Nonverbal vocalizations (NVVs), such as laughter and sighs, convey meaning alongside words. NVV-aware speech systems must recognize and generate these events while preserving lexical accuracy and perceptual quality. Yet differences in annotation, resources, and evaluation complicate comparisons, and aggregate scores can obscure specific weaknesses. We review ten system descriptions from the NVVSpeech Challenge 2026, covering understanding and generation in Chinese and English across 16 event categories. Our analysis examines supervision construction, task adaptation, and inference-time or preference-based feedback. Results reveal persistent recognition difficulties with breath and gasp and trade-offs between event recognition and lexical accuracy, as well as between generation intelligibility and acoustic quality, with human listening providing complementary evaluation evidence. These findings identify priorities for advancing NVV-aware speech understanding, controllable generation, and comparable evaluation.
https://arxiv.org/abs/2610.16076
Existing audio deepfake detection (ADD) methods frequently exhibit limited generalization, as fixed bottleneck architectures fail to adapt to the heterogeneous information densities inherent in diverse spoofing artifacts. Moreover, standard augmentation often leads to a pathological robustness-accuracy trade-off. In this paper, we propose a unified ADD framework to address these challenges. Our approach integrates: (1) an Adaptive Audio Feature Integration (AAFI) module that dynamically selects optimal latent dimensions via a compression pool and a stochastic exploration mechanism, ensuring flexible representations across multiple types of dynamic attacks; and (2) a Mel-Band Adversarial Perturbation (Mel-BAP) strategy that applies targeted regularization in the mel-spectrogram domain to encourage the model to learn invariant decision boundaries. Extensive evaluations on benchmarks including ADD 2022, FakeOrReal, and In-the-Wild demonstrate that our model based on Whisper-small achieves a state-of-the-art average Equal Error Rate (EER) of 1.51%. Compared to existing SOTA models such as the 3-billion-parameter Resemble-Detect-3B-Omni, our framework reduces the parameter count by up to 91.6% while achieving a 60.7% improvement in average EER. In a separate cross-lingual evaluation, the Whisper-base configuration achieves EERs of 3.91-5.65% on five unseen languages. The framework also maintains strong resilience to real-world acoustic distortions. This work provides a scalable and efficient "plug-in" solution for enhancing trust and security in the audio Web ecosystem.
https://arxiv.org/abs/2610.16049
Recent diffusion- and conditional flow matching (CFM)-based emotional voice conversion (EVC) systems achieve strong conversion quality. However, they typically rely on multiple sampling steps, because the conditioning specifies which emotion to impose while its acoustic realization is left to the sampler. In this paper, we propose Prime-EVC, a one-step expressive EVC architecture based on emotion-primed flow matching. Our key idea is to reallocate the emotional conversion between the conditioning and the sampler: a FiLM-fused speaker-emotion style, an emotional variance adaptor, and direct emotion and spectro-temporal supervision assemble an emotion-rich conditioning upstream, rendering the residual conversion left to the CFM decoder attainable within one Euler step. Experiments on the ESD corpus demonstrate that single-step Prime-EVC surpasses five representative EVC systems on every quality metric and exceeds 4- and 100-step DurFlex-EVC in subjective naturalness and emotion while running 3.4--50 times faster than both.
https://arxiv.org/abs/2610.16036
Text-conditioned symbolic music generation has attracted growing interest with the development of generative models. Yet two questions remain underexplored: what aspects of music the conditioning text should describe, and how the corresponding data should be collected and trained to improve controllability and text-music alignment. We address these challenges with TASTE, a taxonomy-aware framework for symbolic music generation with a main focus on piano pieces. First, we introduce a taxonomy of musical elements that specifies how a piece can be captured through textual description in a musically grounded way. The taxonomy comprises a set of metadata tags (genre, emotion, section plan, etc.) together with a free-form description of music imagery. This structure decomposes textual input to match the generation goal more coherently and precisely. Second, we build a data pipeline for collecting, annotating, and retrieving public symbolic piano resources, yielding a corpus of 432K MIDI files labeled at multiple levels of granularity. Finally, we design a staged training procedure that leverages this multi-granularity data across two model families: an autoregressive model and a non-autoregressive diffusion model. Both objective and subjective evaluations show that our models achieve superior musical-element control, text-description alignment, and overall music quality. We will release the full pipeline, all annotated datasets, and the model checkpoints.
https://arxiv.org/abs/2610.15960
Transformer models trained on symbolic music have become remarkably capable. However, it is yet unclear how they represent music structure, and whether understanding their internal representations can be used to unlock methods for model control. We examine this through a case study of harmonic analysis: when a music transformer navigates a cadence, what does it internally "know" about its harmonic context, and can that knowledge be used to affect the generation? Using Aria, a LLaMA-based symbolic music transformer evaluated on Bach chorales, we find that harmonic decisions are causally localized progressively from middle layers onward. Linear probes show that mode becomes decodable in the same layer range where patching identifies growing causal effect, while key identity emerges later. We then confirm these representations are actionable for music generation control: steering via latent directions corresponding to harmonic changes reliably produces a deceptive or minor resolution, while preserving structural coherence and generation quality. This demonstrates that, at inference time, the model's internal harmonic knowledge can be directly manipulated to control the harmonic output.
https://arxiv.org/abs/2610.15957
The Flow Matching objective supervises the velocity field at intermediate states, whereas waveform metrics are evaluated after decoding and may be non-differentiable. To align Flow Matching models with waveform-level and perceptual quality objectives during post-training, FlowGRPO optimizes their trajectories using endpoint rewards. In this work, we focus on music separation and use multi-resolution short-time Fourier transform (MRSTFT), scale-invariant signal-to-distortion ratio (SI-SDR), and Virtual Speech Quality Objective Listener (ViSQOL) as reward components. Before policy optimization, we analyze the relationship between data quality factors and the within-group gap between the maximum and mean rollout rewards for each component. This analysis motivates a revised ViSQOL reward that matches channel-wise root-mean-square (RMS) levels and evaluates the interval from the first to the last active frame, retaining inactive frames within that interval. We also apply a sample-quality filter based on predefined waveform checks. Together, the revised reward and quality filter enable FlowGRPO to improve the pretrained vocal separator in both separation performance and perceptual quality.
https://arxiv.org/abs/2610.15780
Eyewear that senses articulation with inaudible ultrasound can now recognize silent speech over an open vocabulary, but its accuracy is limited by the cost of training data. We present SoniGAN, which synthesizes the ultrasonic echo profiles of silent articulation from text and uses them as synthetic supervision. A voice-cloned text-to-speech (TTS) model reads a text corpus, and a two-stage generator synthesizes echo profiles from representations of the spoken content of that audio. Pretraining a recognizer on 104 hours of the synthesized data improves the word error rate (WER) of a public open-vocabulary benchmark by 2.3 points. Adding the sessions the corpus discards for a faulty microphone brings the system to 22.2% without any language model. Lexicon-constrained decoding, LLM rescoring, and ensembling further bring it to 14.9%, the first minimally obtrusive silent speech interface to fall inside the error range in which users report no loss of willingness to use silent speech input.
https://arxiv.org/abs/2610.15671
We present AnyEdit, a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments, including noise, reverberation, and background music. Recent unified generative models already achieve strong speech and singing generation in clean conditions, but real-world vocal editing remains challenging because the model must revise content while simultaneously preserving speaker identity, prosody, and the surrounding acoustic scene. Directly modeling noisy audio end-to-end entangles semantic editing with environmental acoustics, weakening the core vocal generation capability. We therefore adopt a two-stage design that preserves clean-domain editing ability while deferring acoustic rendering to a separate stage: an instruction-guided autoregressive editor first predicts clean prosody and content-style tokens, supported by a teacher-student noise-robust prosody tokenizer that distills clean prosodic tokens from degraded recordings; a flow-matching acoustic model then performs in-context rendering conditioned on the source acoustic context, so the edited segment inherits surrounding noise, reverberation, and background sound while maintaining consistent speaker timbre. To stabilize unedited regions, we introduce unmodified-area-aware supervision; after pre-training, we further apply direct preference optimization on mined real-to-real singing pairs. We also release AnyEditBench, a benchmark covering both speech and singing voice editing under clean, simulated, and in-the-wild acoustic conditions. Experiments on Chinese and English speech and singing datasets showed that AnyEdit consistently outperforms strong baselines in content accuracy, subjective quality, and background consistency under noisy and reverberant conditions. Demo: this https URL. AnyEditBench: this https URL.
https://arxiv.org/abs/2610.15665
Sleep-disordered breathing, including snoring, hypopnea, and obstructive apnea, is the hallmark of obstructive sleep apnea, whose severity is quantified by the Apnea-Hypopnea Index (AHI). As conventional polysomnography is costly and clin- ically demanding, microphone-only detection has emerged as a noninvasive alternative for screening. We propose ALE- Confidence Fusion Sound Event Detection (ACF-SED), the first framework that directly injects the per-frame confidence map of an Adaptive Line Enhancer (ALE) into Transformer atten- tion biasing and feature modulation, rather than as mere pre- processing. A Multi-Resolution ALE Bank aggregates three parallel NLMS filters at distinct decorrelation delays, while a Confidence-Guided Cross-Path block couples event and noise streams via gated cross-attention. On the Audio-Polygraphy Dataset for Sleep Apnea Analysis, ACF-SED achieves state- of-the-art detection across Event-F1, Segment-F1, and PSDS, enabling end-to-end AHI estimation.
https://arxiv.org/abs/2610.15532
We introduce Non-musical Audio to Symbolic Music Transformation (NASMT), a novel task that converts non-musical audio into symbolic music representations. Unlike automatic music transcription, which assumes musical input and a unique correct score, NASMT is inherently one-to-many and requires evaluation along two axes: input correspondence and musicality. We present Sori, an encoder--decoder Transformer that addresses this task through domain adversarial training (DAT), which bridges the gap between musical and non-musical audio in the encoder's feature space, and classifier-free guidance (CFG) with a continuous parameter~$\gamma$ that controls the trade-off between correspondence and musicality at inference time. Across objective metrics and a listening study, Sori provides effective continuous control via $\gamma$ and compares favorably with instrument-agnostic transcription models. Sori supports real-time streaming on a consumer CPU. Audio examples are available at this https URL.
https://arxiv.org/abs/2610.15474
Neural audio codecs achieve remarkable performance but operate as closed ecosystems with rigidly co-trained encoderdecoder pairs, severely hindering interoperability. To resolve this, we propose BridgeCodec, a versatile framework that decouples these pairs by translating latent representations between mismatched, frozen endpoints. We formulate this cross-codec translation as a Schrödinger Bridge optimal transport problem, enabling robust mapping between disparate distributions without predefined noise priors. To efficiently capture longrange speech temporal dependencies, we integrate a Mambaenhanced U-Net architecture. We validate BridgeCodec on an extreme deployment scenario: mapping a lightweight 8 kHz source encoder directly to a 48 kHz target decoder at an ultralow bitrate of 1 kbps. Experimental results demonstrate that BridgeCodec achieves superior wideband reconstruction and high perceptual quality, successfully bridging heavily bottlenecked, heterogeneous codecs.
https://arxiv.org/abs/2610.15453
Mandarin stuttered speech remains under-resourced, limiting the development of robust speech processing systems. We present a Mandarin stuttering speech synthesis and data augmentation framework that adapts both autoregressive and non-autoregressive TTS models using naturally stuttered speech. Specifically, we adapt CosyVoice2 and MaskGCT to support controllable generation of specific stuttering events while also modeling broader utterance-level characteristics of stuttered speech. Using the adapted models, we construct synthetic data resources for downstream augmentation. Evaluations show that the generated speech closely approaches real stuttered speech. Extensive ASR experiments demonstrate consistent augmentation gains and further observe a domain-gap--diversity trade-off between synthetic and real data. These results demonstrate the value of learning synthetic stuttering data directly from real dysfluency for robust Mandarin speech processing.
https://arxiv.org/abs/2606.21457
Speaker anonymization (SA) aims to conceal speaker identity while preserving linguistic content and emotion, but existing streaming systems often lose emotion information through discrete content bottlenecks or require emotion-specific modeling. We present LiveAnon, a voice-conversion-based streaming SA system that directly conditions an autoregressive codec language model on hidden features from a pretrained streaming ASR encoder, preserving emotion without emotion-specific supervision. To reduce codec-generation latency, we introduce capped RVQ decoding, which preserves dependencies among coarse codebooks while predicting the remaining codebooks in parallel. Under VPC24, LiveAnon improves intelligibility and emotion preservation over fixed-target Stream-Voice-Anon (SVA), a strong prior streaming SA system, matches emotion-prompted SVA in emotion preservation, and achieves the strongest semi-informed privacy among streaming systems. Under VPC26, it ranks second across all metrics among the evaluated anonymization systems, demonstrating a strong privacy-utility balance.
https://arxiv.org/abs/2610.15379
Beat tracking, chord estimation, and structure analysis are staple tasks of music information retrieval. Their datasets are drawn largely from Western popular music, and jazz remains a repertoire on which state-of-the-art systems underperform. We present JSIR, the Jazz Standards dataset for Information Retrieval. Its 267 recordings of 44 jazz standards span 25.6 hours, 3.7 times the audio of JAAH, the closest existing resource. For each recording, we provide annotations for beat and downbeat positions, chords, and sections, all produced by a jazz musician and revised in several passes. In baseline experiments, models trained with JSIR also improve on other jazz datasets: the downbeat F-measure of a state-of-the-art beat tracker rises from 87.7% to 89.2%, and the seventh-chord accuracy of a pop-trained chord estimator on JAAH from 34% to 41%. We publicly release the annotations, the annotation tool used to create them, and a web application that displays them in sync with the audio.
https://arxiv.org/abs/2610.15356
We present Gugak-VocalSet, the first publicly available audio dataset for Korean traditional singing with dense annotations of sigimsae, a structured pitch contour system of vocal expressions. Gugak-VocalSet contains 24.5 hours of unaccompanied solo vocal recordings across 305 tracks of representative repertoire curated by the National Gugak Center, performed by 20 professional singers in a controlled studio environment with fixed key and tempo. In addition to genre, lyrics, and mood/timbre tags, the dataset provides a unified sigimsae ontology and approximately 68,000 segments (71,700 instances including multi-label) of audio-grounded sigimsae annotations; the ontology adopts a functional-position-based hierarchy designed for consistent application across genres, and 14 expert annotators specializing in Korean traditional music performed segment-level labeling accordingly. Baseline experiments on sigimsae classification and event detection show that domain-informed F0 representations remain competitive with, and on classification surpass, frozen probing of a general-purpose music foundation model, while pitch and spectral cues play complementary roles in event-level detection. Companion page: this https URL
https://arxiv.org/abs/2610.15350
Open-vocabulary audio-visual event localization aims to infer explicit segment-level categories for both seen and unseen audio-visual events. Existing solutions rely heavily on hand-crafted prompt templates, whose predictions are highly sensitive to subtle wording variations. Learnable prompts remove this manual tuning burden, but when trained solely on seen events, they tend to overfit to seen categories and distort the inter-event semantic topology required for generalization to unseen categories, a failure we term semantic topology drift. In this paper, we propose SIMPLE, a semantic structure-aligned prompt learning framework that aligns cross-category semantic topology between learnable and hand-crafted prompts across asynchronous audio and visual streams. Specifically, our SIMPLE combines Learnable Continuous Prompt Tuning (LCPT) with Semantic Topology Consistency Regularization (STCR). The LCPT automates prompt engineering through data-driven optimization, while the STCR module aligns the category-wise similarity distributions of learnable and hand-crafted prompts under random category subset sampling to alleviate semantic topology drift. Comprehensive experiments and analyses on the OV-AVEBench demonstrate that our SIMPLE framework achieves state-of-the-art performance on the seen and unseen splits, while using roughly half the trainable parameters and roughly one-third the training time.
https://arxiv.org/abs/2610.15342
Removing audio can shift a visual predictor's output without showing whether the task uses the intended audio information. We introduce Conditional Input and Visual Interpretation Checks (CIViC), a four-gate protocol that links an interpretation to an intervention, held-out task comparison, or endpoint control. We demonstrate CIViC on actor-disjoint face-raster tests, a synthetic audio-feature mixture, and source-video-disjoint SpeakerVid proxy tasks. In the static face-raster test, cross-actor audio replacement meets the registered detection rule with a small effect; on a separate temporal endpoint, constant-audio fits score better than audio-trained fits in four matched seeds. In the synthetic mixture, the condition-augmented fit has higher 1,000-step RMSE than the audio-only fit (0.1807 vs. 0.1707), with the gap shrinking during training. In the SpeakerVid proxy, the five-feature-zero fit (five low-level audio summaries zeroed; identity, dialogue, and condition inputs retained) has lower whole-frame grayscale RMSE than aligned audio across four held-out conditions and five paired seeds, while aligned audio has lower segment-drift error. The results show that output activation, held-out error, and endpoint controls support different interpretations across these proxy tasks.
https://arxiv.org/abs/2610.15327
High-fidelity audio coding requires faithful reconstruction of perceptually important high-frequency components, which are often lost due to the aggressive temporal decimation and quantization of neural codecs. State-of-the-art neural audio coding methods rely on adversarial learning to hallucinate these missing components, yielding perceptually plausible but imprecise reconstructions; a limitation for high-fidelity music applications. As a remedy, we propose high-efficiency neural audio coding (HE-NAC), which provides a multi-band extension of the popular residual vector quantization (RVQ) scheme. HE-NAC introduces an effective training mechanism that harmonizes a bitrate-free blind upsampling process, residual learning to compensate it with minimal bitrate, and a multi-head decoder architecture for band-specific reconstruction. As a result, HE-NAC achieves a balanced trade-off, reliably reconstructing the core-band ($\leq$ 3 kHz) component while preserving overall sound fidelity. Subjective listening tests and objective evaluations demonstrate that HE-NAC outperforms both the DAC baseline and MPEG HE-AAC at equal or lower bitrates.
https://arxiv.org/abs/2610.15243
Currently, voice-related privacy risks are studied under the assumption that the attacker has access only to spoken audio. In contrast, real-world collections often pair the speech signal with side information; yet to our knowledge, no publicly available dataset provides sufficiently detailed and directly linkable speech and side-information pairs to evaluate the combined risk. We propose a distribution-constrained method to build a semi-synthetic dataset and study singling out risk when geographic side information is assumed to accompany anonymized speech. We instantiate it using the English portion of Mozilla Common Voice and the 2021 UK Census, under ignorant and semi-informed attackers, and introduce a conditional reference that isolates the contribution of the audio beyond the candidate-set reduction induced by the side information. In our controlled geographic case study, for single-utterance speakers, the semi-informed attacker reaches an observed isolation probability of 0.718 in the finest OA-plus-household setting, compared with 0.380 in the signal-only setting; its excess over the conditional reference increases from 0.012 to 0.340. The ignorant attacker shows the same trend at a lower level. The effect is largest at fine granularity and short recordings, precisely the setting in which signal-only evaluation looks safest. Our results show that geographic side information can increase the singling out risk of anonymized speech, and we recommend including attacker settings with side information in future studies of voice privacy.
https://arxiv.org/abs/2610.26884
Word-aligned prosody is increasingly used to detect depression and posttraumatic stress disorder (PTSD), yet the stability and clinical relevance of these measurements remain unclear. We examine stability at two levels. At the word level, we ask whether a speaker produces the same word in a similar speaker-specific way across different utterance contexts. At the speaker level, we ask whether averaging prosody across many words produces consistent speaker scores across separate recordings. Drawing on principles of psychological measurement, we compare these representations in four corpora spanning acted, spontaneous, and clinical speech. Word-specific deviations show little cross-context recurrence (0.01-0.16), whereas speaker summaries are highly consistent, reaching 0.997 in ESD, 0.73-0.90 in MSP-Podcast, and 0.84-0.996 in DAIC-WOZ and PTSD-STOP. Despite this consistency, associations with depression or PTSD severity do not exceed 0.15; equivalence tests exclude effects of 0.30 in all analyses and 0.20 in DAIC-WOZ, while within-person associations with daily PTSD symptoms are near zero ($|\rho|\leq0.04$). Thus, aggregation yields stable speaker-level prosodic measures, but these measures do not detectably track depression or PTSD severity in the studied corpora. Code is available at this https URL.
https://arxiv.org/abs/2610.15181