mlsanityData to 14 Oct 2026

Attention as a Guide for Simultaneous Speech Translation

arXiv 2212.07850 ↗First posted 15 Dec 2022cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

The study of the attention mechanism has sparked interest in many fields, such as language modeling and machine translation. Although its patterns have been exploited to perform different tasks, from neural network understanding to textual alignment, no previous work has analysed the encoder-decoder attention behavior in speech translation (ST) nor used it to improve ST on a specific task. In this paper, we fill this gap by proposing an attention-based policy (EDAtt) for simultaneous ST (SimulST) that is motivated by an analysis of the existing attention relations between audio input and textual output. Its goal is to leverage the encoder-decoder attention scores to guide inference in real time. Results on en->{de, es} show that the EDAtt policy achieves overall better results compared to the SimulST state of the art, especially in terms of computational-aware latency.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 56 of its 86 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

56 of 86 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  9. 2016Sequence-Level Knowledge Distillationcited by 801
  10. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  11. 2015Listen, Attend and Spellcited by 543
  12. 2019Revealing the Dark Secrets of BERTcited by 301
38 more
  1. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  2. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  3. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 156
  4. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 151
  5. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  6. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  7. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  8. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 104
  9. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  10. 2019Monotonic Multihead Attentioncited by 93
  11. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 88
  12. 2017Online and Linear-Time Attention by Enforcing Monotonic Alignmentscited by 85
  13. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 81
  14. 2019Jointly Learning to Align and Translate with Transformer Modelscited by 75
  15. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 57
  16. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  17. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 49
  18. 2021CTC-based Compression for Direct Speech Translationcited by 45
  19. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
  20. 2020Simultaneous Translation Policies: From Fixed to Adaptivecited by 43
  21. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  22. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 40
  23. 2022Information-Transport-based Policy for Simultaneous Translationcited by 40
  24. 2020Accurate Word Alignment Induction from Neural Machine Translationcited by 32
  25. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  26. 2019Adding Interpretable Attention to Neural Translation Models Improves Word Alignmentcited by 30
  27. 2020Streaming Simultaneous Speech Translation with Augmented Memory Transformercited by 30
  28. 2022CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022cited by 30
  29. 2019Thinking Slow about Latency Evaluation for Simultaneous Machine Translationcited by 28
  30. 2022Does Simultaneous Speech Translation need Simultaneous Models?cited by 22
  31. 2018An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translationcited by 15
  32. 2021Speechformer: Reducing Information Loss in Direct Speech Translationcited by 15
  33. 2021An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategiescited by 14
  34. 2022Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translationcited by 14
  35. 2022Efficient yet Competitive Speech Translation: FBK@IWSLT2022cited by 14
  36. 2021The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021cited by 13
  37. 2020On Target Segmentation for Direct Speech Translationcited by 12
  38. 2016Guided Alignment Training for Topic-Aware Neural Machine Translationcited by 11

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 16; 2025: 9; 2026: 4.

32 papers.

Sort:

By area:

  1. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  2. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  3. 2024TransLLaMa: LLM-based Simultaneous Translation Systemcited by 34
  4. 2024StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learningcited by 21
  5. 2024StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selectioncited by 19
  6. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  7. 2024Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agentcited by 14
  8. 2025Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecturecited by 12
  9. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  10. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  11. 2024Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detectioncited by 7
  12. 2024FASST: Fast LLM-based Simultaneous Speech Translationcited by 7
20 more
  1. 2023Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023cited by 6
  2. 2025StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Modelcited by 6
  3. 2024Recent Advances in End-to-End Simultaneous Speech Translationcited by 5
  4. 2024Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translationcited by 5
  5. 2025Large Language Models Are Read/Write Policy-Makers for Simultaneous Generationcited by 5
  6. 2024Overcoming Non-monotonicity in Transducer-based Streaming Generationcited by 4
  7. 2024Simultaneous Interpretation Corpus Construction by Large Language Models in Distant Language Paircited by 3
  8. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3
  9. 2025SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translationcited by 2
  10. 2025SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translationcited by 2
  11. 2024SBAAM! Eliminating Transcript Dependency in Automatic Subtitlingcited by 1
  12. 2024SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translationcited by 1
  13. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  14. 2026Streaming Translation and Transcription Through Speech-to-Text Causal Alignmentcited by 1
  15. 2026DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMscited by 1
  16. 2024CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translationcited by 0
  17. 2025Overcoming Latency Bottlenecks in On-Device Speech Translation: A Cascaded Approach with Alignment-Based Streaming MTcited by 0
  18. 2025Direct Simultaneous Translation Activation for Large Audio-Language Modelscited by 0
  19. 2026X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation Systemcited by 0
  20. 2026Attention-Based Adaptive Policies for Simultaneous Speech-to-Text Translationcited by 0