mlsanityData to 14 Oct 2026

Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation

arXiv 2406.04541 ↗First posted 6 Jun 2024cs.CL eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

While the neural transducer is popular for online speech recognition, simultaneous speech translation (SST) requires both streaming and re-ordering capabilities. This paper presents the LS-Transducer-SST, a label-synchronous neural transducer for SST, which naturally possesses these two properties. The LS-Transducer-SST dynamically decides when to emit translation tokens based on an Auto-regressive Integrate-and-Fire (AIF) mechanism. A latency-controllable AIF is also proposed, which can control the quality-latency trade-off either only during decoding, or it can be used in both decoding and training. The LS-Transducer-SST can naturally utilise monolingual text-only data via its prediction network which helps alleviate the key issue of data sparsity for E2E SST. During decoding, a chunk-based incremental joint decoding technique is designed to refine and expand the search space. Experiments on the Fisher-CallHome Spanish (Es-En) and MuST-C En-De data show that the LS-Transducer-SST gives a better quality-latency trade-off than existing popular methods. For example, the LS-Transducer-SST gives a 3.1/2.9 point BLEU increase (Es-En/En-De) relative to CAAT at a similar latency and a 1.4 s reduction in average lagging latency with similar BLEU scores relative to Wait-k.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

34 of 45 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  5. 2016Sequence-Level Knowledge Distillationcited by 801
  6. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  7. 2015Attention-Based Models for Speech Recognitioncited by 401
  8. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  9. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  10. 2019Monotonic Multihead Attentioncited by 93
  11. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  12. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 81
22 more
  1. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 77
  2. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  3. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 63
  4. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 57
  5. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  6. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 49
  7. 2020Simultaneous Translation Policies: From Fixed to Adaptivecited by 43
  8. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 40
  9. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  10. 2021Factorized Neural Transducer for Efficient Language Model Adaptationcited by 32
  11. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  12. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  13. 2021Learning When to Translate for Streaming Speechcited by 30
  14. 2021Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translationcited by 23
  15. 2022Does Simultaneous Speech Translation need Simultaneous Models?cited by 22
  16. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  17. 2022Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translationcited by 14
  18. 2022Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translationcited by 11
  19. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10
  20. 2021Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediatescited by 9
  21. 2023Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoffcited by 8
  22. 2023Label-Synchronous Neural Transducer for End-to-End ASRcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  2. 2024Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognitioncited by 4
  3. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3
  4. 2025SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translationcited by 2
  5. 2025SimulSense: Sense-Driven Interpreting for Efficient Simultaneous Speech Translationcited by 0