# BabyLM's First Words: Word Segmentation as a Phonological Probing Task

arXiv: https://arxiv.org/abs/2504.03338 · HTML: https://mlsanity.com/paper/2504.03338/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 71 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Byte Latent Transformer: Patches Scale Better Than Tokens](https://mlsanity.com/paper/2412.09871/) (2024)
- [Character-Level Language Modeling with Deeper Self-Attention](https://mlsanity.com/paper/1808.04444/) (2018)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner](https://mlsanity.com/paper/1607.08723/) (2016)
- [Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2301.11796/) (2023)
- [Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2412.05149/) (2024)
- [The Zero Resource Speech Challenge 2021: Spoken language modelling](https://mlsanity.com/paper/2104.14700/) (2021)
- [Analyzing Cognitive Plausibility of Subword Tokenization](https://mlsanity.com/paper/2310.13348/) (2023)
- [BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models](https://mlsanity.com/paper/2306.01506/) (2023)
- [Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate](https://mlsanity.com/paper/1807.04783/) (2018)
- [Phonotactic Complexity and its Trade-offs](https://mlsanity.com/paper/2005.03774/) (2020)
- [Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions](https://mlsanity.com/paper/2301.08810/) (2023)
- [From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes](https://mlsanity.com/paper/2410.22906/) (2024)
- [The statistical trade-off between word order and word structure - large-scale evidence for the principle of least effort](https://mlsanity.com/paper/1608.03587/) (2016)
- [Are word boundaries useful for unsupervised language learning?](https://mlsanity.com/paper/2210.02956/) (2022)
- [On the Difficulty of Segmenting Words with Attention](https://mlsanity.com/paper/2109.10107/) (2021)
- [Subword models struggle with word learning, but surprisal hides it](https://mlsanity.com/paper/2502.12835/) (2025)
- [IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modeling](https://mlsanity.com/paper/2504.03036/) (2025)
- [Tabula nearly rasa: Probing the Linguistic Knowledge of Character-Level Neural Language Models Trained on Unsegmented Text](https://mlsanity.com/paper/1906.07285/) (2019)
- [Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamas](https://mlsanity.com/paper/2410.01487/) (2024)

## Built on by (most cited first, 2 of 2)

- [PRiSM: Benchmarking Phone Realization in Speech Models](https://mlsanity.com/paper/2601.14046/) (2026)
- [Cross-Linguistic Effects in Bilingual Phoneme BabyLMs](https://mlsanity.com/paper/2609.37121/) (2026)

## Abstract (arXiv)

Language models provide a key framework for studying linguistic theories based on prediction, but phonological analysis using large language models (LLMs) is difficult; there are few phonological benchmarks beyond English and the standard input representation used in LLMs (subwords of graphemes) is not suitable for analyzing the representation of phonemes. In this work, we demonstrate how word segmentation can be used as a phonological probing task, allowing us to study the representations learned by phoneme-based language models trained on child-directed speech across 31 languages. Following computational models of word segmentation, we present unsupervised methods for extracting word boundaries from a trained model using the observation that prediction-error peaks at the start of words. We also use linear probes to identify that these models implicitly track word boundaries, even when they do not appear in training. This cross-lingual work corroborates statistical learning theories of acquisition and empirically motivates new methods for training subword tokenizers.