mlsanityData to 14 Oct 2026

AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation

arXiv 2305.11408 ↗First posted 19 May 2023cs.CL cs.LG cs.SDcited by 35

Not on the map. Why?

Abstract (arXiv)

Attention is the core mechanism of today's most used architectures for natural language processing and has been analyzed from many perspectives, including its effectiveness for machine translation-related tasks. Among these studies, attention resulted to be a useful source of information to get insights about word alignment also when the input text is substituted with audio segments, as in the case of the speech translation (ST) task. In this paper, we propose AlignAtt, a novel policy for simultaneous ST (SimulST) that exploits the attention information to generate source-target alignments that guide the model during inference. Through experiments on the 8 language pairs of MuST-C v1.0, we show that AlignAtt outperforms previous state-of-the-art SimulST policies applied to offline-trained models with gains in terms of BLEU of 2 points and latency reductions ranging from 0.5s to 0.8s across the 8 languages.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

32 of 43 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  7. 2016Sequence-Level Knowledge Distillationcited by 801
  8. 2015Listen, Attend and Spellcited by 543
  9. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  10. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  11. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 88
  12. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
20 more
  1. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 81
  2. 2019Jointly Learning to Align and Translate with Transformer Modelscited by 75
  3. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  4. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  5. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 57
  6. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 54
  7. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 49
  8. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  9. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  10. 2021CTC-based Compression for Direct Speech Translationcited by 45
  11. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 40
  12. 2020Accurate Word Alignment Induction from Neural Machine Translationcited by 32
  13. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  14. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  15. 2022CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022cited by 30
  16. 2022Does Simultaneous Speech Translation need Simultaneous Models?cited by 22
  17. 2021Regularizing End-to-End Speech Translation with Triangular Decomposition Agreementcited by 16
  18. 2018An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translationcited by 15
  19. 2021An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategiescited by 14
  20. 2022Defining maximum acceptable latency of AI-enhanced CAI toolscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 12; 2026: 8.

35 papers.

Sort:

By area:

  1. 2024StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selectioncited by 19
  2. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  3. 2025Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecturecited by 12
  4. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  5. 2025InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Modelcited by 8
  6. 2025Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025cited by 8
  7. 2024Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detectioncited by 7
  8. 2024FASST: Fast LLM-based Simultaneous Speech Translationcited by 7
  9. 2023Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023cited by 6
  10. 2024Recent Advances in End-to-End Simultaneous Speech Translationcited by 5
  11. 2024Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translationcited by 5
  12. 2024Overcoming Non-monotonicity in Transducer-based Streaming Generationcited by 4
23 more
  1. 2025Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systemscited by 4
  2. 2024Simultaneous Interpretation Corpus Construction by Large Language Models in Distant Language Paircited by 3
  3. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3
  4. 2025Locate-and-Focus: Enhancing Terminology Translation in Speech Language Modelscited by 3
  5. 2025MLLP-VRAIN UPV system for the IWSLT 2025 Simultaneous Speech Translation Translation taskcited by 2
  6. 2025SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translationcited by 2
  7. 2025SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translationcited by 2
  8. 2024Word Order in English-Japanese Simultaneous Interpretation: Analyses and Evaluation using Chunk-wise Monotonic Translationcited by 1
  9. 2024SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translationcited by 1
  10. 2024NAIST Simultaneous Speech Translation System for IWSLT 2024cited by 1
  11. 2024CMU's IWSLT 2024 Simultaneous Speech Translation Systemcited by 1
  12. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  13. 2026Streaming Translation and Transcription Through Speech-to-Text Causal Alignmentcited by 1
  14. 2026DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMscited by 1
  15. 2024CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translationcited by 0
  16. 2025SimulSense: Sense-Driven Interpreting for Efficient Simultaneous Speech Translationcited by 0
  17. 2025OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusioncited by 0
  18. 2026SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translationcited by 0
  19. 2026Streaming Speech-to-Text Translation with a SpeechLLMcited by 0
  20. 2026A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026cited by 0
  21. 2026AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Taskcited by 0
  22. 2026X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation Systemcited by 0
  23. 2026Attention-Based Adaptive Policies for Simultaneous Speech-to-Text Translationcited by 0