# Around the world in 60 words: A generative vocabulary test for online research

arXiv: https://arxiv.org/abs/2302.01614 · HTML: https://mlsanity.com/paper/2302.01614/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 21 references that are in the map.
- First posted on arXiv 3 Feb 2023.

## Builds on (most cited first, 3 of 3)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Gibbs Sampling with People](https://mlsanity.com/paper/2008.02595/) (2020)

## Built on by (most cited first, 1 of 1)

- [Establishing Vocabulary Tests as a Benchmark for Evaluating Large Language Models](https://mlsanity.com/paper/2310.14703/) (2023)

## Abstract (arXiv)

Conducting experiments with diverse participants in their native languages can uncover insights into culture, cognition, and language that may not be revealed otherwise. However, conducting these experiments online makes it difficult to validate self-reported language proficiency. Furthermore, existing proficiency tests are small and cover only a few languages. We present an automated pipeline to generate vocabulary tests using text from Wikipedia. Our pipeline samples rare nouns and creates pseudowords with the same low-level statistics. Six behavioral experiments (N=236) in six countries and eight languages show that (a) our test can distinguish between native speakers of closely related languages, (b) the test is reliable (r=0.82), and (c) performance strongly correlates with existing tests (LexTale) and self-reports. We further show that test accuracy is negatively correlated with the linguistic distance between the tested and the native language. Our test, available in eight languages, can easily be extended to other languages.