# CMU's IWSLT 2024 Simultaneous Speech Translation System

arXiv: https://arxiv.org/abs/2408.07452 · HTML: https://mlsanity.com/paper/2408.07452/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 14 Aug 2024.

## Builds on (most cited first, 11 of 11)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation](https://mlsanity.com/paper/2206.05807/) (2022)
- [AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2305.11408/) (2023)
- [Speech Translation with Large Language Models: An Industrial Practice](https://mlsanity.com/paper/2312.13585/) (2023)
- [On the Impact of Noises in Crowd-Sourced Data for Speech Translation](https://mlsanity.com/paper/2206.13756/) (2022)

## Built on by (most cited first, 1 of 1)

- [EMMeTT: Efficient Multimodal Machine Translation Training](https://mlsanity.com/paper/2409.13523/) (2024)

## Abstract (arXiv)

This paper describes CMU's submission to the IWSLT 2024 Simultaneous Speech Translation (SST) task for translating English speech to German text in a streaming manner. Our end-to-end speech-to-text (ST) system integrates the WavLM speech encoder, a modality adapter, and the Llama2-7B-Base model as the decoder. We employ a two-stage training approach: initially, we align the representations of speech and text, followed by full fine-tuning. Both stages are trained on MuST-c v2 data with cross-entropy loss. We adapt our offline ST model for SST using a simple fixed hold-n policy. Experiments show that our model obtains an offline BLEU score of 31.1 and a BLEU score of 29.5 under 2 seconds latency on the MuST-C-v2 tst-COMMON.