# SBAAM! Eliminating Transcript Dependency in Automatic Subtitling

arXiv: https://arxiv.org/abs/2405.10741 · HTML: https://mlsanity.com/paper/2405.10741/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 84 references that are in the map.
- First posted on arXiv 17 May 2024.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [On decoder-only architecture for speech-to-text and large language model integration](https://mlsanity.com/paper/2307.03917/) (2023)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates](https://mlsanity.com/paper/1911.03167/) (2019)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [Speech Translation and the End-to-End Promise: Taking Stock of Where We Are](https://mlsanity.com/paper/2004.06358/) (2020)
- [CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition](https://mlsanity.com/paper/2007.09127/) (2020)
- [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019)
- [Bridging the Modality Gap for Speech-to-Text Translation](https://mlsanity.com/paper/2010.14920/) (2020)
- [Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders](https://mlsanity.com/paper/2105.05752/) (2021)
- [CTC-based Compression for Direct Speech Translation](https://mlsanity.com/paper/2102.01578/) (2021)
- [SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training](https://mlsanity.com/paper/2210.03730/) (2022)
- [Multilingual End-to-End Speech Translation](https://mlsanity.com/paper/1910.00254/) (2019)
- [Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information](https://mlsanity.com/paper/1911.02215/) (2019)
- [Accurate Word Alignment Induction from Neural Machine Translation](https://mlsanity.com/paper/2004.14837/) (2020)
- [Attention as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2212.07850/) (2022)

## Built on by (most cited first, 1 of 1)

- [Enhancing Entertainment Translation for Indian Languages using Adaptive Context, Style and LLMs](https://mlsanity.com/paper/2412.20440/) (2024)

## Abstract (arXiv)

Subtitling plays a crucial role in enhancing the accessibility of audiovisual content and encompasses three primary subtasks: translating spoken dialogue, segmenting translations into concise textual units, and estimating timestamps that govern their on-screen duration. Past attempts to automate this process rely, to varying degrees, on automatic transcripts, employed diversely for the three subtasks. In response to the acknowledged limitations associated with this reliance on transcripts, recent research has shifted towards transcription-free solutions for translation and segmentation, leaving the direct generation of timestamps as uncharted territory. To fill this gap, we introduce the first direct model capable of producing automatic subtitles, entirely eliminating any dependence on intermediate transcripts also for timestamp prediction. Experimental results, backed by manual evaluation, showcase our solution's new state-of-the-art performance across multiple language pairs and diverse conditions.