mlsanityData to 7 Oct 2026

Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent

arXiv 2407.21646 ↗First posted 31 Jul 2024cs.CL cs.SD eess.AScited by 14

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

In this paper, we present Cross Language Agent -- Simultaneous Interpretation, CLASI, a high-quality and human-like Simultaneous Speech Translation (SiST) System. Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency. To address the challenge of translating in-domain terminologies, CLASI employs a multi-modal retrieving module to obtain relevant information to augment the translation. Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information. Experimental results show that our system outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate CLASI with a better human evaluation metric, valid information proportion (VIP), which measures the amount of information that can be successfully conveyed to the listeners. In the real-world scenarios, where the speeches are often disfluent, informal, and unclear, CLASI achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese translation directions, respectively. In contrast, state-of-the-art commercial or open-source systems only achieve 35.4% and 41.6%. On the extremely hard dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70% VIP.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 53 of its 82 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

53 of 82 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Survey on Large Language Model based Autonomous Agents22 Aug

Other references · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
38 more
  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  3. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  4. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  5. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  6. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  7. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  8. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  9. 2016Learning to Translate in Real-time with Neural Machine Translationcited by 97
  10. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  11. 2016Can neural machine translation do simultaneous translation?cited by 85
  12. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 80
  13. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  14. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 70
  15. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  16. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 56
  17. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
  18. 2014Addressing the Rare Word Problem in Neural Machine Translationcited by 50
  19. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 47
  20. 2023Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decodercited by 44
  21. 2024Fine-tuning Large Language Models for Domain-specific Machine Translationcited by 43
  22. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 39
  23. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  24. 2021Learning When to Translate for Streaming Speechcited by 30
  25. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  26. 2023End-to-End Simultaneous Speech Translation with Differentiable Segmentationcited by 22
  27. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  28. 2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
  29. 2024StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learningcited by 20
  30. 2021BSTC: A Large-Scale Chinese-English Speech Translation Datasetcited by 16
  31. 2021An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategiescited by 14
  32. 2024MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translationcited by 10
  33. 2022MT Metrics Correlate with Human Ratings of Simultaneous Speech Translationcited by 9
  34. 2016Neural Name Translation Improves Neural Machine Translationcited by 8
  35. 2023Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoffcited by 8
  36. 2022Continuous Rating as Reliable Human Evaluation of Simultaneous Speech Translationcited by 6
  37. 2023Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023cited by 6
  38. 2023Simul-LLM: A Framework for Exploring High-Quality Simultaneous Translation with Large Language Modelscited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 4.

14 papers.

Sort:

By area:

  1. 2025LLM Agents for Education: Advances and Applicationscited by 59
  2. 2025SeqPO-SiMT: Sequential Policy Optimization for Simultaneous Machine Translationcited by 13
  3. 2025Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voicecited by 9
  4. 2025StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Modelcited by 6
  5. 2025LLMs Can Achieve High-quality Simultaneous Machine Translation as Efficiently as Offlinecited by 5
  6. 2025Spatial Speech Translation: Translating Across Space With Binaural Hearablescited by 3
  7. 2025PHRASED: Phrase Dictionary Biasing for Speech Translationcited by 1
  8. 2026Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speechcited by 1
  9. 2025From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognitioncited by 0
  10. 2025Toward Machine Interpreting: Lessons from Human Interpreting Studiescited by 0
  11. 2025SimulSense: Sense-Driven Interpreting for Efficient Simultaneous Speech Translationcited by 0
  12. 2026Regularized Entropy Information Adaptation with Temporal-Awareness Networks for Simultaneous Speech Translationcited by 0
2 more
  1. 2026Bridging the Usability Gap: Lessons from Interpreting Studies for Machine Interpreting Designcited by 0
  2. 2026EviSI: An Evidence-Based Evaluation Agent for Simultaneous Interpretingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Translating the Translator: Decomposing the Cost of English-Forced Inter-Agent Communication
  2. 2023Speech Translation with Large Language Models: An Industrial Practice
  3. 2026One Voice, Many Tongues: Cross-Lingual Voice Cloning for Scientific Speech
  4. 2024Agent-SiMT: Agent-assisted Simultaneous Machine Translation with Large Language Models
  5. 2026AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task
  6. 2024Investigating Decoder-only Large Language Models for Speech-to-text Translation
  7. 2026KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs
  8. 2026Loong: A Human-Like Long Document Translation Agent with Observe-and-Act Adaptive Context Selection