# SubER: A Metric for Automatic Evaluation of Subtitle Quality

arXiv: https://arxiv.org/abs/2205.05805 · HTML: https://mlsanity.com/paper/2205.05805/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 24 references that are in the map.
- First posted on arXiv 11 May 2022.

## Builds on (most cited first, 5 of 5)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Is 42 the Answer to Everything in Subtitling-oriented Speech Translation?](https://mlsanity.com/paper/2006.01080/) (2020)
- [MuST-Cinema: a Speech-to-Subtitles corpus](https://mlsanity.com/paper/2002.10829/) (2020)

## Built on by (most cited first, 9 of 9)

- [Fine-tuning Whisper on Low-Resource Languages for Real-World Applications](https://mlsanity.com/paper/2412.15726/) (2024)
- [Direct Speech Translation for Automatic Subtitling](https://mlsanity.com/paper/2209.13192/) (2022)
- [Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023](https://mlsanity.com/paper/2309.15554/) (2023)
- [EVE: Towards End-to-End Video Subtitle Extraction with Vision-Language Models](https://mlsanity.com/paper/2503.04058/) (2025)
- [SBAAM! Eliminating Transcript Dependency in Automatic Subtitling](https://mlsanity.com/paper/2405.10741/) (2024)
- [Optimizing Estonian TV Subtitles with Semi-supervised Learning and LLMs](https://mlsanity.com/paper/2501.05234/) (2025)
- [V-SAT: Video Subtitle Annotation Tool](https://mlsanity.com/paper/2510.24180/) (2025)
- [From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs](https://mlsanity.com/paper/2512.19161/) (2025)
- [Breaking Babel: A Self-Evolving Multi-Agent System for Long-Form Subtitle Translation](https://mlsanity.com/paper/2609.38660/) (2026)

## Abstract (arXiv)

This paper addresses the problem of evaluating the quality of automatically generated subtitles, which includes not only the quality of the machine-transcribed or translated speech, but also the quality of line segmentation and subtitle timing. We propose SubER - a single novel metric based on edit distance with shifts that takes all of these subtitle properties into account. We compare it to existing metrics for evaluating transcription, translation, and subtitle quality. A careful human evaluation in a post-editing scenario shows that the new metric has a high correlation with the post-editing effort and direct human assessment scores, outperforming baseline metrics considering only the subtitle text, such as WER and BLEU, and existing methods to integrate segmentation and timing features.