# Who Are We Talking About? Handling Person Names in Speech Translation

arXiv: https://arxiv.org/abs/2205.06755 · HTML: https://mlsanity.com/paper/2205.06755/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 13 May 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates](https://mlsanity.com/paper/1911.03167/) (2019)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [Tied Multitask Learning for Neural Speech Translation](https://mlsanity.com/paper/1802.06655/) (2018)
- [Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?](https://mlsanity.com/paper/2106.01045/) (2021)
- [Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systems](https://mlsanity.com/paper/2011.11564/) (2020)
- [Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET](https://mlsanity.com/paper/2202.05148/) (2022)
- [Consistent Transcription and Translation of Speech](https://mlsanity.com/paper/2007.12741/) (2020)
- [Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation](https://mlsanity.com/paper/2005.10678/) (2020)
- [On Target Segmentation for Direct Speech Translation](https://mlsanity.com/paper/2009.04707/) (2020)
- [End-to-end named entity extraction from speech](https://mlsanity.com/paper/1805.12045/) (2018)
- [Is "moby dick" a Whale or a Bird? Named Entities and Terminology in Speech Translation](https://mlsanity.com/paper/2109.07439/) (2021)
- [Towards the evaluation of automatic simultaneous speech translation from a communicative perspective](https://mlsanity.com/paper/2103.08364/) (2021)
- [Automatic Estimation of Simultaneous Interpreter Performance](https://mlsanity.com/paper/1805.04016/) (2018)
- [KUDO Interpreter Assist: Automated Real-time Support for Remote Interpretation](https://mlsanity.com/paper/2201.01800/) (2022)

## Built on by (most cited first, 8 of 8)

- [Recent Advances in Direct Speech-to-text Translation](https://mlsanity.com/paper/2306.11646/) (2023)
- [Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments](https://mlsanity.com/paper/2307.03354/) (2023)
- [Joint Speech Translation and Named Entity Recognition](https://mlsanity.com/paper/2210.11987/) (2022)
- [Named Entity Detection and Injection for Direct Speech Translation](https://mlsanity.com/paper/2210.11981/) (2022)
- [Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach](https://mlsanity.com/paper/2409.09009/) (2024)
- [Robustness of Multi-Source MT to Transcription Errors](https://mlsanity.com/paper/2305.16894/) (2023)
- [PHRASED: Phrase Dictionary Biasing for Speech Translation](https://mlsanity.com/paper/2506.09175/) (2025)
- [LOGIC: Efficient and Robust Contextual Biasing for Speech LLMs via Logit-Space Integration](https://mlsanity.com/paper/2601.15397/) (2026)

## Abstract (arXiv)

Recent work has shown that systems for speech translation (ST) -- similarly to automatic speech recognition (ASR) -- poorly handle person names. This shortcoming does not only lead to errors that can seriously distort the meaning of the input, but also hinders the adoption of such systems in application scenarios (like computer-assisted interpreting) where the translation of named entities, like person names, is crucial. In this paper, we first analyse the outputs of ASR/ST systems to identify the reasons of failures in person name transcription/translation. Besides the frequency in the training data, we pinpoint the nationality of the referred person as a key factor. We then mitigate the problem by creating multilingual models, and further improve our ST systems by forcing them to jointly generate transcripts and translations, prioritising the former over the latter. Overall, our solutions result in a relative improvement in token-level person name accuracy by 47.8% on average for three language pairs (en->es,fr,it).