# R-BI: Regularized Batched Inputs enhance Incremental Decoding Framework for Low-Latency Simultaneous Speech Translation

arXiv: https://arxiv.org/abs/2401.05700 · HTML: https://mlsanity.com/paper/2401.05700/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019)
- [Learning to Translate in Real-time with Neural Machine Translation](https://mlsanity.com/paper/1610.00388/) (2016)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [Monotonic Multihead Attention](https://mlsanity.com/paper/1909.12406/) (2019)
- [SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2011.02048/) (2020)
- [STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation](https://mlsanity.com/paper/2203.10426/) (2022)
- [Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation](https://mlsanity.com/paper/1806.03661/) (2018)
- [Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection](https://mlsanity.com/paper/2005.11185/) (2020)
- [WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2203.15455/) (2022)
- [Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition](https://mlsanity.com/paper/2012.05481/) (2020)
- [Re-translation versus Streaming for Simultaneous Translation](https://mlsanity.com/paper/2004.03643/) (2020)
- [CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022](https://mlsanity.com/paper/2204.06028/) (2022)
- [Modeling Dual Read/Write Paths for Simultaneous Machine Translation](https://mlsanity.com/paper/2203.09163/) (2022)
- [Super-Human Performance in Online Low-latency Recognition of Conversational Speech](https://mlsanity.com/paper/2010.03449/) (2020)
- [The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021](https://mlsanity.com/paper/2107.00279/) (2021)
- [Cascaded Models With Cyclic Feedback For Direct Speech Translation](https://mlsanity.com/paper/2010.11153/) (2020)
- [Learning Hard Alignments with Variational Inference](https://mlsanity.com/paper/1705.05524/) (2017)

## Built on by (most cited first, 1 of 1)

- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)

## Abstract (arXiv)

Incremental Decoding is an effective framework that enables the use of an offline model in a simultaneous setting without modifying the original model, making it suitable for Low-Latency Simultaneous Speech Translation. However, this framework may introduce errors when the system outputs from incomplete input. To reduce these output errors, several strategies such as Hold-n, LA-n, and SP-n can be employed, but the hyper-parameter n needs to be carefully selected for optimal performance. Moreover, these strategies are more suitable for end-to-end systems than cascade systems. In our paper, we propose a new adaptable and efficient policy named "Regularized Batched Inputs". Our method stands out by enhancing input diversity to mitigate output errors. We suggest particular regularization techniques for both end-to-end and cascade systems. We conducted experiments on IWSLT Simultaneous Speech Translation (SimulST) tasks, which demonstrate that our approach achieves low latency while maintaining no more than 2 BLEU points loss compared to offline systems. Furthermore, our SimulST systems attained several new state-of-the-art results in various language directions.