# Lhotse: a speech data representation library for the modern deep learning ecosystem

arXiv: https://arxiv.org/abs/2110.12561 · HTML: https://mlsanity.com/paper/2110.12561/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2021.

## Built on by (most cited first, 30 of 33)

- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST](https://mlsanity.com/paper/2509.14128/) (2025)
- [Granary: Speech Recognition and Translation Dataset in 25 European Languages](https://mlsanity.com/paper/2505.13404/) (2025)
- [CR-CTC: Consistency regularization on CTC for improved speech recognition](https://mlsanity.com/paper/2410.05101/) (2024)
- [Fast and parallel decoding for transducer](https://mlsanity.com/paper/2211.00484/) (2022)
- [The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources](https://mlsanity.com/paper/2406.16746/) (2024)
- [Updated Corpora and Benchmarks for Long-Form Speech Recognition](https://mlsanity.com/paper/2309.15013/) (2023)
- [Speech collage: code-switched audio generation by collaging monolingual corpora](https://mlsanity.com/paper/2309.15674/) (2023)
- [Blank-regularized CTC for Frame Skipping in Neural Transducer](https://mlsanity.com/paper/2305.11558/) (2023)
- [Training and Inference Efficiency of Encoder-Decoder Speech Models](https://mlsanity.com/paper/2503.05931/) (2025)
- [Delay-penalized transducer for low-latency streaming ASR](https://mlsanity.com/paper/2211.00490/) (2022)
- [VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning](https://mlsanity.com/paper/2410.17485/) (2024)
- [Word Level Timestamp Generation for Automatic Speech Recognition and Translation](https://mlsanity.com/paper/2505.15646/) (2025)
- [TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding](https://mlsanity.com/paper/2601.06896/) (2026)
- [EMMeTT: Efficient Multimodal Machine Translation Training](https://mlsanity.com/paper/2409.13523/) (2024)
- [Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation](https://mlsanity.com/paper/2511.16757/) (2025)
- [Efficient Data Selection for Domain Adaptation of ASR Using Pseudo-Labels and Multi-Stage Filtering](https://mlsanity.com/paper/2506.03681/) (2025)
- [VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents](https://mlsanity.com/paper/2608.13831/) (2026)
- [Better Semi-supervised Learning for Multi-domain ASR Through Incremental Retraining and Data Filtering](https://mlsanity.com/paper/2506.04981/) (2025)
- [Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition](https://mlsanity.com/paper/2510.03723/) (2025)
- [CLiFT-ASR: A Cross-Lingual Fine-Tuning Framework for Low-Resource Taiwanese Hokkien Speech Recognition](https://mlsanity.com/paper/2511.06860/) (2025)
- [Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper](https://mlsanity.com/paper/2409.13499/) (2024)
- [HENT-SRT: Hierarchical Efficient Neural Transducer with Self-Distillation for Joint Speech Recognition and Translation](https://mlsanity.com/paper/2506.02157/) (2025)
- [Towards Comprehensive Semantic Speech Embeddings for Chinese Dialects](https://mlsanity.com/paper/2601.07274/) (2026)
- [Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methods](https://mlsanity.com/paper/2603.25767/) (2026)
- [A Native-Reference Coordinate Geometry for L2 Pronunciation Deviation Using Self-Supervised Speech Models](https://mlsanity.com/paper/2609.28060/) (2026)
- [MooseNet: A Trainable Metric for Synthesized Speech with a PLDA Module](https://mlsanity.com/paper/2301.07087/) (2023)
- [Pitch-Aware RNN-T for Mandarin Chinese Mispronunciation Detection and Diagnosis](https://mlsanity.com/paper/2406.04595/) (2024)
- [Less is More: Accurate Speech Recognition & Translation without Web-Scale Data](https://mlsanity.com/paper/2406.19674/) (2024)
- [ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration](https://mlsanity.com/paper/2409.09506/) (2024)

## Abstract (arXiv)

Speech data is notoriously difficult to work with due to a variety of codecs, lengths of recordings, and meta-data formats. We present Lhotse, a speech data representation library that draws upon lessons learned from Kaldi speech recognition toolkit and brings its concepts into the modern deep learning ecosystem. Lhotse provides a common JSON description format with corresponding Python classes and data preparation recipes for over 30 popular speech corpora. Various datasets can be easily combined together and re-purposed for different tasks. The library handles multi-channel recordings, long recordings, local and cloud storage, lazy and on-the-fly operations amongst other features. We introduce Cut and CutSet concepts, which simplify common data wrangling tasks for audio and help incorporate acoustic context of speech utterances. Finally, we show how Lhotse leverages PyTorch data API abstractions and adopts them to handle speech data for deep learning.