# Wav2Gloss: Generating Interlinear Glossed Text from Speech

arXiv: https://arxiv.org/abs/2403.13169 · HTML: https://mlsanity.com/paper/2403.13169/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 65 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 25 of 25)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer](https://mlsanity.com/paper/2401.16658/) (2024)
- [Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling](https://mlsanity.com/paper/1810.03459/) (2018)
- [Improving Massively Multilingual ASR With Auxiliary CTC Objectives](https://mlsanity.com/paper/2302.12829/) (2023)
- [SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2210.08634/) (2022)
- [Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages](https://mlsanity.com/paper/2305.12606/) (2023)
- [Leveraging End-to-End ASR for Endangered Language Documentation: An Empirical Study on Yolox'ochitl Mixtec](https://mlsanity.com/paper/2101.10877/) (2021)
- [DARTS-ASR: Differentiable Architecture Search for Multilingual Speech Recognition and Adaptation](https://mlsanity.com/paper/2005.07029/) (2020)
- [Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond](https://mlsanity.com/paper/2310.05513/) (2023)
- [Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learning](https://mlsanity.com/paper/2306.15686/) (2023)
- [When Is TTS Augmentation Through a Pivot Language Useful?](https://mlsanity.com/paper/2207.09889/) (2022)

## Built on by (most cited first, 9 of 9)

- [POWSM: A Phonetic Open Whisper-Style Speech Foundation Model](https://mlsanity.com/paper/2510.24992/) (2025)
- [Interdisciplinary Research in Conversation: A Case Study in Computational Morphology for Language Documentation](https://mlsanity.com/paper/2509.10644/) (2025)
- [Massively Multilingual Joint Segmentation and Glossing](https://mlsanity.com/paper/2601.10925/) (2026)
- [ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration](https://mlsanity.com/paper/2409.09506/) (2024)
- [Boosting the Capabilities of Compact Models in Low-Data Contexts with Large Language Models and Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.00387/) (2024)
- [Morphologically-Informed Tokenizers for Languages with Non-Concatenative Morphology: A case study of Yolox'ochtil Mixtec ASR](https://mlsanity.com/paper/2512.06169/) (2025)
- [Hybrid Neural-LLM Pipeline for Morphological Glossing in Endangered Language Documentation: A Case Study of Jungar Tuvan](https://mlsanity.com/paper/2603.00923/) (2026)
- [CWoMP: Morpheme Representation Learning for Interlinear Glossing](https://mlsanity.com/paper/2603.18184/) (2026)
- [Phone Segmentation and Recognition through Phonological Activation Mapping](https://mlsanity.com/paper/2607.09020/) (2026)

## Abstract (arXiv)

Thousands of the world's languages are in danger of extinction--a tremendous threat to cultural identities and human language diversity. Interlinear Glossed Text (IGT) is a form of linguistic annotation that can support documentation and resource creation for these languages' communities. IGT typically consists of (1) transcriptions, (2) morphological segmentation, (3) glosses, and (4) free translations to a majority language. We propose Wav2Gloss: a task in which these four annotation components are extracted automatically from speech, and introduce the first dataset to this end, Fieldwork: a corpus of speech with all these annotations, derived from the work of field linguists, covering 37 languages, with standard formatting, and train/dev/test splits. We provide various baselines to lay the groundwork for future research on IGT generation from speech, such as end-to-end versus cascaded, monolingual versus multilingual, and single-task versus multi-task approaches.