# On the Use of External Data for Spoken Named Entity Recognition

arXiv: https://arxiv.org/abs/2112.07648 · HTML: https://mlsanity.com/paper/2112.07648/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 75 references that are in the map.
- First posted on arXiv 14 Dec 2021.

## Builds on (most cited first, 30 of 43)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Dice Loss for Data-imbalanced NLP Tasks](https://mlsanity.com/paper/1911.02855/) (2019)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)

## Built on by (most cited first, 12 of 12)

- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks](https://mlsanity.com/paper/2212.10525/) (2022)
- [Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages](https://mlsanity.com/paper/2205.01086/) (2022)
- [A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding](https://mlsanity.com/paper/2211.05869/) (2022)
- [End-to-end model for named entity recognition from speech without paired training data](https://mlsanity.com/paper/2204.00803/) (2022)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Context-aware Fine-tuning of Self-supervised Speech Models](https://mlsanity.com/paper/2212.08542/) (2022)
- [Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training](https://mlsanity.com/paper/2305.12793/) (2023)
- [Integrating Pretrained ASR and LM to Perform Sequence Generation for Spoken Language Understanding](https://mlsanity.com/paper/2307.11005/) (2023)
- [Few-Shot Spoken Language Understanding via Joint Speech-Text Models](https://mlsanity.com/paper/2310.05919/) (2023)
- [GRASS: Unified Generation Model for Speech-to-Semantic Tasks](https://mlsanity.com/paper/2309.02780/) (2023)
- [Joint Automatic Speech Recognition And Structure Learning For Better Speech Understanding](https://mlsanity.com/paper/2501.07329/) (2025)

## Abstract (arXiv)

Spoken language understanding (SLU) tasks involve mapping from speech audio signals to semantic labels. Given the complexity of such tasks, good performance might be expected to require large labeled datasets, which are difficult to collect for each new task and domain. However, recent advances in self-supervised speech representations have made it feasible to consider learning SLU models with limited labeled data. In this work we focus on low-resource spoken named entity recognition (NER) and address the question: Beyond self-supervised pre-training, how can we use external speech and/or text data that are not annotated for the task? We draw on a variety of approaches, including self-training, knowledge distillation, and transfer learning, and consider their applicability to both end-to-end models and pipeline (speech recognition followed by text NER model) approaches. We find that several of these approaches improve performance in resource-constrained settings beyond the benefits from pre-trained representations alone. Compared to prior work, we find improved F1 scores of up to 16%. While the best baseline model is a pipeline approach, the best performance when using external data is ultimately achieved by an end-to-end model. We provide detailed comparisons and analyses, showing for example that end-to-end models are able to focus on the more NER-specific words.