mlsanityData to 14 Oct 2026

RASST: Retrieval-Augmented Simultaneous Speech Translation

arXiv 2601.22777 ↗First posted 30 Jan 2026cs.CLcited by 0

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Simultaneous speech translation produces target text incrementally from partial speech input. Recent speech large language models have markedly improved SST quality but still struggle with rare and domain-specific terminology. Retrieval augmentation has helped in automatic speech recognition and neural machine translation, but extending it to SST is non-trivial: retrieval must be fast and accurate under partial speech, and the model must decide whether and when to apply retrieved terms during incremental generation. We propose Retrieval-Augmented Simultaneous Speech Translation (RASST), which addresses both challenges. For accurate cross-modal retrieval under partial input, RASST trains a lightweight speech-text retriever that produces chunkwise terminology hints for the Speech LLM via multi-scale retrieval. To use these hints correctly, we synthesize training data that teaches the Speech LLM to decide whether and when to apply each retrieved term. Experiments on ACL 60/60 dev set and the ESO test set show that RASST improves terminology accuracy by nearly 40% and overall translation quality by up to 3 BLEU points, with negligible computational overhead.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 33 of its 55 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

33 of 55 references.

References · 33

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  6. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  7. 2020Nearest Neighbor Machine Translationcited by 209
  8. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 156
  9. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  10. 2018Deep context: end-to-end contextual speech recognitioncited by 137
  11. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 101
  12. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
21 more
  1. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 93
  2. 2020Contextual RNN-T For Open Domain ASRcited by 85
  3. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 81
  4. 2020Deep Shallow Fusion for RNN-T Personalizationcited by 64
  5. 2018Guiding Neural Machine Translation with Retrieved Translation Piecescited by 63
  6. 2024TransLLaMa: LLM-based Simultaneous Translation Systemcited by 34
  7. 2021Instant One-Shot Word-Learning for Context-Specific Neural Sequence-to-Sequence Speech Recognitioncited by 30
  8. 2021Learning Kernel-Smoothed Machine Translation with Retrieved Examplescited by 25
  9. 2021Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognitioncited by 25
  10. 2024StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selectioncited by 19
  11. 2024CTC-Assisted LLM-Based Contextual ASRcited by 19
  12. 2025Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecturecited by 12
  13. 2025BR-ASR: Efficient and Scalable Bias Retrieval Framework for Contextual Biasing ASR in Speech LLMcited by 11
  14. 2023Retrieve and Copy: Scaling ASR Personalization to Large Catalogscited by 7
  15. 2023INK: Injecting kNN Knowledge in Nearest Neighbor Machine Translationcited by 6
  16. 2025StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Modelcited by 6
  17. 2024Deferred NAM: Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASRcited by 5
  18. 2023On-the-fly Text Retrieval for End-to-End ASR Adaptationcited by 4
  19. 2025Leveraging Domain Knowledge at Inference Time for LLM Translation: Retrieval versus Generationcited by 4
  20. 2024Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approachcited by 3
  21. 2025Locate-and-Focus: Enhancing Terminology Translation in Speech Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Retrieval Augmented End-to-End Spoken Dialog Models
  2. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing
  3. 2025SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation
  4. 2024Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison
  5. 2024Retrieval-Augmented Machine Translation with Unstructured Knowledge
  6. 2024Optimizing example selection for retrieval-augmented machine translation with translation memories
  7. 2025Improving Retrieval-Augmented Neural Machine Translation with Monolingual Data
  8. 2023Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding