mlsanityData to 14 Oct 2026

Quieter Than the Room: Representation Drift and Task Robustness in Speech Encoders

arXiv 2609.27195 ↗First posted 23 Sep 2026cs.SD cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Non-speech interference can change a speech representation without causing comparable task loss. We test eight frozen encoders on four tasks, adding non-speech sounds throughout recordings, during speech, or in pauses. Under whole-recording interference, embedding drift tracks task loss across seven sounds, with mean Spearman correlations of 0.81-0.88. Moving the same sound between speech and pauses changes this pattern. At quiet to moderate levels, pause interference produces larger drift, while speech interference usually causes greater loss on intent recognition, speaker verification and speech recognition. Emotion recognition shows a weaker placement effect. Pause interference also changes speech-frame representations beyond the injected region. Even below the estimated recording background, interference can change embeddings as much as repeated speech takes do. Drift helps rank the effects of different sounds, but larger drift does not consistently indicate greater task loss.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 26 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

20 of 26 references.

References · 20

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  8. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  9. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  10. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 132
  11. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  12. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
8 more
  1. 2024SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metricscited by 38
  2. 2025Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audiocited by 31
  3. 2020A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differencescited by 23
  4. 2024Speech Robust Bench: A Robustness Benchmark For Speech Recognitioncited by 21
  5. 2025Effective Context in Neural Speech Modelscited by 5
  6. 2026Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Modelscited by 2
  7. 2025Self-Supervised Speech Quality Assessment (S3QA): Leveraging Speech Foundation Models for a Scalable Speech Quality Metriccited by 1
  8. 2026SEAM: Shortcut-Aware Real-Time Detection of Scripted vs. Spontaneous Speech for Interview Guardrailscited by 1

Built on by

Not cited yet.