# On the Difficulty of Segmenting Words with Attention

arXiv: https://arxiv.org/abs/2109.10107 · HTML: https://mlsanity.com/paper/2109.10107/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 21 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Professor Forcing: A New Algorithm for Training Recurrent Networks](https://mlsanity.com/paper/1610.09038/) (2016)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech](https://mlsanity.com/paper/1803.08976/) (2018)
- [The Zero Resource Speech Challenge 2017](https://mlsanity.com/paper/1712.04313/) (2017)
- [Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces](https://mlsanity.com/paper/1805.07467/) (2018)
- [A segmental framework for fully-unsupervised large-vocabulary speech recognition](https://mlsanity.com/paper/1606.06950/) (2016)
- [End-to-End Neural Segmental Models for Speech Recognition](https://mlsanity.com/paper/1708.00531/) (2017)
- [Unsupervised Word Segmentation from Speech with Attention](https://mlsanity.com/paper/1806.06734/) (2018)
- [Unwritten Languages Demand Attention Too! Word Discovery with Encoder-Decoder Models](https://mlsanity.com/paper/1709.05631/) (2017)
- [Empirical Evaluation of Sequence-to-Sequence Models for Word Discovery in Low-resource Settings](https://mlsanity.com/paper/1907.00184/) (2019)
- [NMTPY: A Flexible Toolkit for Advanced Neural Machine Translation Systems](https://mlsanity.com/paper/1706.00457/) (2017)
- [Acoustic-to-Word Recognition with Sequence-to-Sequence Models](https://mlsanity.com/paper/1807.09597/) (2018)

## Built on by (most cited first, 6 of 6)

- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring](https://mlsanity.com/paper/2202.11929/) (2022)
- [Keyword localisation in untranscribed speech using visually grounded speech models](https://mlsanity.com/paper/2202.01107/) (2022)
- [Whisper Has an Internal Word Aligner](https://mlsanity.com/paper/2509.09987/) (2025)
- [Supervised Attention in Sequence-to-Sequence Models for Speech Recognition](https://mlsanity.com/paper/2204.12308/) (2022)
- [BabyLM's First Words: Word Segmentation as a Phonological Probing Task](https://mlsanity.com/paper/2504.03338/) (2025)

## Abstract (arXiv)

Word segmentation, the problem of finding word boundaries in speech, is of interest for a range of tasks. Previous papers have suggested that for sequence-to-sequence models trained on tasks such as speech translation or speech recognition, attention can be used to locate and segment the words. We show, however, that even on monolingual data this approach is brittle. In our experiments with different input types, data sizes, and segmentation algorithms, only models trained to predict phones from words succeed in the task. Models trained to predict words from either phones or speech (i.e., the opposite direction needed to generalize to new data), yield much worse results, suggesting that attention-based segmentation is only useful in limited scenarios.