# End-to-end Named Entity Recognition from English Speech

arXiv: https://arxiv.org/abs/2005.11184 · HTML: https://mlsanity.com/paper/2005.11184/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2020.

## Built on by (most cited first, 30 of 36)

- [SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech](https://mlsanity.com/paper/2111.10367/) (2021)
- [SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks](https://mlsanity.com/paper/2212.10525/) (2022)
- [Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages](https://mlsanity.com/paper/2205.01086/) (2022)
- [AISHELL-NER: Named Entity Recognition from Chinese Speech](https://mlsanity.com/paper/2202.08533/) (2022)
- [DeToxy: A Large-Scale Multimodal Dataset for Toxicity Classification in Spoken Utterances](https://mlsanity.com/paper/2110.07592/) (2021)
- [CopyNE: Better Contextual ASR by Copying Named Entities](https://mlsanity.com/paper/2305.12839/) (2023)
- [On the Use of External Data for Spoken Named Entity Recognition](https://mlsanity.com/paper/2112.07648/) (2021)
- [Is "moby dick" a Whale or a Bird? Named Entities and Terminology in Speech Translation](https://mlsanity.com/paper/2109.07439/) (2021)
- [PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations](https://mlsanity.com/paper/2203.16965/) (2022)
- [Failing Forward: Improving Generative Error Correction for ASR with Synthetic Data and Retrieval Augmentation](https://mlsanity.com/paper/2410.13198/) (2024)
- [Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialect](https://mlsanity.com/paper/2407.04533/) (2024)
- [Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speech](https://mlsanity.com/paper/2105.04719/) (2021)
- [Using Large Language Model for End-to-End Chinese ASR and NER](https://mlsanity.com/paper/2401.11382/) (2024)
- [Joint Speech Translation and Named Entity Recognition](https://mlsanity.com/paper/2210.11987/) (2022)
- [Visual Information Matters for ASR Error Correction](https://mlsanity.com/paper/2303.10160/) (2023)
- [Improving Code-Switching and Named Entity Recognition in ASR with Speech Editing based Data Augmentation](https://mlsanity.com/paper/2306.08588/) (2023)
- [TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR](https://mlsanity.com/paper/2407.04444/) (2024)
- ["I've Heard of You!": Generate Spoken Named Entity Recognition Data for Unseen Entities](https://mlsanity.com/paper/2412.19102/) (2024)
- [Medical Spoken Named Entity Recognition](https://mlsanity.com/paper/2406.13337/) (2024)
- [wav2graph: A Framework for Supervised Learning Knowledge Graph from Speech](https://mlsanity.com/paper/2408.04174/) (2024)
- [PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding](https://mlsanity.com/paper/2506.11064/) (2025)
- [NE-PADD: Leveraging Named Entity Knowledge for Robust Partial Audio Deepfake Detection via Attention Aggregation](https://mlsanity.com/paper/2509.03829/) (2025)
- [SpeechLLM: Unified Speech and Language Model for Enhanced Multi-Task Understanding in Low Resource Settings](https://mlsanity.com/paper/2509.04473/) (2025)
- [Cross-stitched Multi-modal Encoders](https://mlsanity.com/paper/2204.09227/) (2022)
- [Mining Word Boundaries in Speech as Naturally Annotated Word Segmentation Data](https://mlsanity.com/paper/2210.17122/) (2022)
- [E2E Spoken Entity Extraction for Virtual Agents](https://mlsanity.com/paper/2302.10186/) (2023)
- [Leveraging Cross-Lingual Transfer Learning in Spoken Named Entity Recognition Systems](https://mlsanity.com/paper/2307.01310/) (2023)
- [CPPF: A contextual and post-processing-free model for automatic speech recognition](https://mlsanity.com/paper/2309.07413/) (2023)
- [Mining Word Boundaries from Speech-Text Parallel Data for Cross-domain Chinese Word Segmentation](https://mlsanity.com/paper/2412.09045/) (2024)
- [Joint Automatic Speech Recognition And Structure Learning For Better Speech Understanding](https://mlsanity.com/paper/2501.07329/) (2025)

## Abstract (arXiv)

Named entity recognition (NER) from text has been a widely studied problem and usually extracts semantic information from text. Until now, NER from speech is mostly studied in a two-step pipeline process that includes first applying an automatic speech recognition (ASR) system on an audio sample and then passing the predicted transcript to a NER tagger. In such cases, the error does not propagate from one step to another as both the tasks are not optimized in an end-to-end (E2E) fashion. Recent studies confirm that integrated approaches (e.g., E2E ASR) outperform sequential ones (e.g., phoneme based ASR). In this paper, we introduce a first publicly available NER annotated dataset for English speech and present an E2E approach, which jointly optimizes the ASR and NER tagger components. Experimental results show that the proposed E2E approach outperforms the classical two-step approach. We also discuss how NER from speech can be used to handle out of vocabulary (OOV) words in an ASR system.