mlsanityData to 7 Oct 2026

Joint Speech Translation and Named Entity Recognition

arXiv 2210.11987 ↗First posted 21 Oct 2022cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Modern automatic translation systems aim at place the human at the center by providing contextual support and knowledge. In this context, a critical task is enriching the output with information regarding the mentioned entities, which is currently achieved processing the generated translation with named entity recognition (NER) and entity linking systems. In light of the recent promising results shown by direct speech translation (ST) models and the known weaknesses of cascades (error propagation and additional latency), in this paper we propose multitask models that jointly perform ST and NER, and compare them with a cascade baseline. The experimental results show that our models significantly outperform the cascade on the NER task (by 0.4-1.0 F1), without degradation in terms of translation quality, and with the same computational efficiency of a plain direct ST model.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

25 of 39 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  8. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  9. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  10. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  11. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  12. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 80
13 more
  1. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 56
  2. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  3. 2021CTC-based Compression for Direct Speech Translationcited by 45
  4. 2020Simultaneous Translation Policies: From Fixed to Adaptivecited by 42
  5. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  6. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 39
  7. 2020End-to-end Named Entity Recognition from English Speechcited by 36
  8. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  9. 2022Does Simultaneous Speech Translation need Simultaneous Models?cited by 22
  10. 2022AISHELL-NER: Named Entity Recognition from Chinese Speechcited by 19
  11. 2022Non-Parametric Domain Adaptation for End-to-End Speech Translationcited by 12
  12. 2021Is "moby dick" a Whale or a Bird? Named Entities and Terminology in Speech Translationcited by 11
  13. 2022Who Are We Talking About? Handling Person Names in Speech Translationcited by 8

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 2.

5 papers.

  1. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  2. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  3. 2024Using Large Language Model for End-to-End Chinese ASR and NERcited by 6
  4. 2025PHRASED: Phrase Dictionary Biasing for Speech Translationcited by 1
  5. 2025Whispering Context: Distilling Syntax and Semantics for Long Speech Transcriptscited by 0