# End-to-end model for named entity recognition from speech without paired training data

arXiv: https://arxiv.org/abs/2204.00803 · HTML: https://mlsanity.com/paper/2204.00803/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 2 Apr 2022.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech](https://mlsanity.com/paper/2104.11462/) (2021)
- [Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2010.04284/) (2020)
- [Curriculum-based transfer learning for an effective end-to-end spoken language understanding and domain portability](https://mlsanity.com/paper/1906.07601/) (2019)
- [You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation](https://mlsanity.com/paper/2005.07157/) (2020)
- [Using Speech Synthesis to Train End-to-End Spoken Language Understanding Models](https://mlsanity.com/paper/1910.09463/) (2019)
- [On the Use of External Data for Spoken Named Entity Recognition](https://mlsanity.com/paper/2112.07648/) (2021)
- [Large-scale Transfer Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2008.05671/) (2020)

## Built on by (most cited first, 11 of 11)

- [AfriNames: Most ASR models "butcher" African Names](https://mlsanity.com/paper/2306.00253/) (2023)
- [Using Large Language Model for End-to-End Chinese ASR and NER](https://mlsanity.com/paper/2401.11382/) (2024)
- [ProGRes: Prompted Generative Rescoring on ASR n-Best](https://mlsanity.com/paper/2409.00217/) (2024)
- [SENSE models: an open source solution for multilingual and multimodal semantic-based tasks](https://mlsanity.com/paper/2509.12093/) (2025)
- [OLISIA: a Cascade System for Spoken Dialogue State Tracking](https://mlsanity.com/paper/2304.11073/) (2023)
- [Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training](https://mlsanity.com/paper/2305.12793/) (2023)
- [Pantagruel: Unified Self-Supervised Encoders for French Text and Speech](https://mlsanity.com/paper/2601.05911/) (2026)
- [Few-Shot Spoken Language Understanding via Joint Speech-Text Models](https://mlsanity.com/paper/2310.05919/) (2023)
- [Leveraging Cross-Lingual Transfer Learning in Spoken Named Entity Recognition Systems](https://mlsanity.com/paper/2307.01310/) (2023)
- [GRASS: Unified Generation Model for Speech-to-Semantic Tasks](https://mlsanity.com/paper/2309.02780/) (2023)
- [CV-18 NER: Augmented Common Voice for Named Entity Recognition from Arabic Speech](https://mlsanity.com/paper/2604.02209/) (2026)

## Abstract (arXiv)

Recent works showed that end-to-end neural approaches tend to become very popular for spoken language understanding (SLU). Through the term end-to-end, one considers the use of a single model optimized to extract semantic information directly from the speech signal. A major issue for such models is the lack of paired audio and textual data with semantic annotation. In this paper, we propose an approach to build an end-to-end neural model to extract semantic information in a scenario in which zero paired audio data is available. Our approach is based on the use of an external model trained to generate a sequence of vectorial representations from text. These representations mimic the hidden representations that could be generated inside an end-to-end automatic speech recognition (ASR) model by processing a speech signal. An SLU neural module is then trained using these representations as input and the annotated text as output. Last, the SLU module replaces the top layers of the ASR model to achieve the construction of the end-to-end model. Our experiments on named entity recognition, carried out on the QUAERO corpus, show that this approach is very promising, getting better results than a comparable cascade approach or than the use of synthetic voices.