mlsanityData to 14 Oct 2026

Overcoming Non-monotonicity in Transducer-based Streaming Generation

arXiv 2411.17170 ↗First posted 26 Nov 2024cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Streaming generation models are utilized across fields, with the Transducer architecture being popular in industrial applications. However, its input-synchronous decoding mechanism presents challenges in tasks requiring non-monotonic alignments, such as simultaneous translation. In this research, we address this issue by integrating Transducer's decoding with the history of input stream via a learnable monotonic attention. Our approach leverages the forward-backward algorithm to infer the posterior probability of alignments between the predictor states and input timestamps, which is then used to estimate the monotonic context representations, thereby avoiding the need to enumerate the exponentially large alignment space during training. Extensive experiments show that our MonoAttn-Transducer effectively handles non-monotonic alignments in streaming scenarios, offering a robust solution for complex generation tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

38 of 49 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  6. 2018Self-Attention with Relative Position Representationscited by 1,120
  7. 2016Sequence-Level Knowledge Distillationcited by 801
  8. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  9. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  10. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 113
  11. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 104
  12. 2016Learning to Translate in Real-time with Neural Machine Translationcited by 97
26 more
  1. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  2. 2016Can neural machine translation do simultaneous translation?cited by 86
  3. 2017Online and Linear-Time Attention by Enforcing Monotonic Alignmentscited by 85
  4. 2017Monotonic Chunkwise Attentioncited by 83
  5. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 81
  6. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  7. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 57
  8. 2021RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformercited by 40
  9. 2020Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memorycited by 35
  10. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  11. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  12. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  13. 2021Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translationcited by 23
  14. 2024StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learningcited by 21
  15. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  16. 2022Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translationcited by 16
  17. 2019Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Frameworkcited by 15
  18. 2024Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Timecited by 15
  19. 2023Non-autoregressive Streaming Transformer for Simultaneous Translationcited by 12
  20. 2023Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translationcited by 11
  21. 2022LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducerscited by 10
  22. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  23. 2021Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attentioncited by 9
  24. 2024StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversioncited by 9
  25. 2023Unified Segment-to-Segment Framework for Simultaneous Sequence Generationcited by 6
  26. 2024Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokenscited by 5

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
  2. 2025StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Modelcited by 6
  3. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3
  4. 2025Efficient Speech Language Modeling via Energy Distance in Continuous Latent Spacecited by 1