mlsanityData to 14 Oct 2026

FASST: Fast LLM-based Simultaneous Speech Translation

arXiv 2408.09430 ↗First posted 18 Aug 2024cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Simultaneous speech translation (SST) takes streaming speech input and generates text translation on the fly. Existing methods either have high latency due to recomputation of input representations, or fall behind of offline ST in translation quality. In this paper, we propose FASST, a fast large language model based method for streaming speech translation. We propose blockwise-causal speech encoding and consistency mask, so that streaming speech input can be encoded incrementally without recomputation. Furthermore, we develop a two-stage training strategy to optimize FASST for simultaneous inference. We evaluate FASST and multiple strong prior models on MuST-C dataset. Experiment results show that FASST achieves the best quality-latency trade-off. It outperforms the previous best model by an average of 1.5 BLEU under the same latency for English to Spanish translation.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 18 Aug 2024.

Builds on

36 of 47 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  9. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  10. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  11. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 145
  12. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
24 more
  1. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 81
  2. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  3. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 57
  4. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 49
  5. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 40
  6. 2022Information-Transport-based Policy for Simultaneous Translationcited by 40
  7. 2020Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memorycited by 35
  8. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  9. 2024TransLLaMa: LLM-based Simultaneous Translation Systemcited by 34
  10. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  11. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  12. 2020Streaming Simultaneous Speech Translation with Augmented Memory Transformercited by 30
  13. 2021Learning When to Translate for Streaming Speechcited by 30
  14. 2023Speech Translation with Large Language Models: An Industrial Practicecited by 28
  15. 2023End-to-End Simultaneous Speech Translation with Differentiable Segmentationcited by 22
  16. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  17. 2024SiLLM: Large Language Models for Simultaneous Machine Translationcited by 13
  18. 2024Simultaneous Masking, Not Prompting Optimization: A Paradigm Shift in Fine-tuning LLMs for Simultaneous Translationcited by 11
  19. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  20. 2023Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoffcited by 8
  21. 2020Temporally Correlated Task Scheduling for Sequence Learningcited by 6
  22. 2023Shiftable Context: Addressing Training-Inference Context Mismatch in Simultaneous Speech Translationcited by 4
  23. 2023Implicit Memory Transformer for Computationally Efficient Simultaneous Speech Translationcited by 4
  24. 2022AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 4.

7 papers.

  1. 2025Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecturecited by 12
  2. 2025InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Modelcited by 8
  3. 2026DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMscited by 1
  4. 2024CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translationcited by 0
  5. 2026Streaming Speech-to-Text Translation with a SpeechLLMcited by 0
  6. 2026Do LLMs Need Architectural Changes for Simultaneous Speech Translation? A Prefix-to-Prefix Data Driven Approachcited by 0
  7. 2026WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMscited by 0