mlsanityData to 14 Oct 2026

Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers

arXiv 2307.03183 ↗First posted 6 Jul 2023cs.SD eess.AScited by 72

Not on the map. Why?

Abstract (arXiv)

In this paper, we focus on Whisper, a recent automatic speech recognition model trained with a massive 680k hour labeled speech corpus recorded in diverse conditions. We first show an interesting finding that while Whisper is very robust against real-world background sounds (e.g., music), its audio representation is actually not noise-invariant, but is instead highly correlated to non-speech sounds, indicating that Whisper recognizes speech conditioned on the noise type. With this finding, we build a unified audio tagging and speech recognition model Whisper-AT by freezing the backbone of Whisper, and training a lightweight audio tagging model on top of it. With <1% extra computational cost, Whisper-AT can recognize audio events, in addition to spoken text, in a single forward pass.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Jul 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 23; 2025: 24; 2026: 15.

72 papers.

Sort:

By area:

  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  2. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  3. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  4. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  5. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  6. 2023Joint Audio and Speech Understandingcited by 100
  7. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 68
  8. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 42
  9. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 31
  10. 2023LLark: A Multimodal Instruction-Following Language Model for Musiccited by 29
  11. 2023Kid-Whisper: Towards Bridging the Performance Gap in Automatic Speech Recognition for Children VS. Adultscited by 27
  12. 2024Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speechcited by 18
38 more
  1. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  2. 2024Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encodingcited by 14
  3. 2025SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuningcited by 14
  4. 2025POWSM: A Phonetic Open Whisper-Style Speech Foundation Modelcited by 14
  5. 2026GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Taskscited by 14
  6. 2025Beyond Transcription: Mechanistic Interpretability in ASRcited by 13
  7. 2024Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whispercited by 12
  8. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  9. 2023Investigating the Emergent Audio Classification Ability of ASR Foundation Modelscited by 10
  10. 2024Crossmodal ASR Error Correction with Discrete Speech Unitscited by 9
  11. 2024On the Evaluation of Speech Foundation Models for Spoken Language Understandingcited by 9
  12. 2024Audio Dialogues: Dialogues dataset for audio and music understandingcited by 7
  13. 2024Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detectioncited by 7
  14. 2026Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadoxcited by 7
  15. 2024Speech foundation models on intelligibility prediction for hearing-impaired listenerscited by 6
  16. 2025Adaptability of ASR Models on Low-Resource Language: A Comparative Study of Whisper and Wav2Vec-BERT on Banglacited by 6
  17. 2024Keyword-Guided Adaptation of Automatic Speech Recognitioncited by 5
  18. 2025DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Modelscited by 5
  19. 2023A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognitioncited by 4
  20. 2024Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCRcited by 4
  21. 2025mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognitioncited by 4
  22. 2025Demographic Attributes Prediction from Speech Using WavLM Embeddingscited by 4
  23. 2025Spatial Audio Processing with Large Language Model on Wearable Devicescited by 4
  24. 2025StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMscited by 4
  25. 2025TICL+: A Case Study On Speech In-Context Learning for Children's Speech Recognitioncited by 4
  26. 2024Who Said What? An Automated Approach to Analyzing Speech in Preschool Classroomscited by 3
  27. 2025Unveiling the Best Practices for Applying Speech Foundation Models to Speech Intelligibility Prediction for Hearing-Impaired Peoplecited by 3
  28. 2025PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMscited by 3
  29. 2025LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Modelcited by 3
  30. 2026Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audiocited by 3
  31. 2023Big model only for hard audios: Sample dependent Whisper model selection for efficient inferencescited by 2
  32. 2024Factor-Conditioned Speaking-Style Captioningcited by 2
  33. 2025Deepfake Detection of Singing Voices With Whisper Encodingscited by 2
  34. 2025Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Modelcited by 2
  35. 2025Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Datacited by 2
  36. 2025Probing the Hidden Talent of ASR Foundation Models for L2 English Oral Assessmentcited by 2
  37. 2026On the Interpretability of Whisper Encodings Using Sparse Autoencoderscited by 2
  38. 2024Transfer Learning from Whisper for Microscopic Intelligibility Predictioncited by 1

50 most cited of 72 All 72 (JSON)