# Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary?

arXiv: https://arxiv.org/abs/2410.03240 · HTML: https://mlsanity.com/paper/2410.03240/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 74 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 13 of 13)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity](https://mlsanity.com/paper/2003.04866/) (2020)
- [PySBD: Pragmatic Sentence Boundary Disambiguation](https://mlsanity.com/paper/2010.09657/) (2020)
- [JTubeSpeech: corpus of Japanese speech collected from YouTube for speech recognition and speaker verification](https://mlsanity.com/paper/2112.09323/) (2021)
- [Lexical Simplification Benchmarks for English, Portuguese, and Spanish](https://mlsanity.com/paper/2209.05301/) (2022)
- [Construction of a Large-scale Japanese ASR Corpus on TV Recordings](https://mlsanity.com/paper/2103.14736/) (2021)
- [Word Familiarity and Frequency](https://mlsanity.com/paper/1806.03431/) (2018)

## Built on by (most cited first, 5 of 5)

- [BQA: Body Language Question Answering Dataset for Video Large Language Models](https://mlsanity.com/paper/2410.13206/) (2024)
- [Towards Automated Lexicography: Generating and Evaluating Definitions for Learner's Dictionaries](https://mlsanity.com/paper/2601.01842/) (2026)
- [Dispersion Measures as Predictors of Lexical Decision Time, Word Familiarity, and Lexical Complexity](https://mlsanity.com/paper/2501.06536/) (2025)
- [Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models](https://mlsanity.com/paper/2511.21086/) (2025)
- [Sakura at BEA 2026 Shared Task 1: What Makes Vocabulary Difficult?](https://mlsanity.com/paper/2605.14257/) (2026)

## Abstract (arXiv)

Word frequency is a key variable in psycholinguistics, useful for modeling human familiarity with words even in the era of large language models (LLMs). Frequency in film subtitles has proved to be a particularly good approximation of everyday language exposure. For many languages, however, film subtitles are not easily available, or are overwhelmingly translated from English. We demonstrate that frequencies extracted from carefully processed YouTube subtitles provide an approximation comparable to, and often better than, the best currently available resources. Moreover, they are available for languages for which a high-quality subtitle or speech corpus does not exist. We use YouTube subtitles to construct frequency norms for five diverse languages, Chinese, English, Indonesian, Japanese, and Spanish, and evaluate their correlation with lexical decision time, word familiarity, and lexical complexity. In addition to being strongly correlated with two psycholinguistic variables, a simple linear regression on the new frequencies achieves a new high score on a lexical complexity prediction task in English and Japanese, surpassing both models trained on film subtitle frequencies and the LLM GPT-4. Our code, the frequency lists, fastText word embeddings, and statistical language models are freely available at https://github.com/naist-nlp/tubelex.