# PuzzLing Machines: A Challenge on Learning From Small Data

arXiv: https://arxiv.org/abs/2004.13161 · HTML: https://mlsanity.com/paper/2004.13161/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2020.

## Built on by (most cited first, 15 of 15)

- [Large Linguistic Models: Investigating LLMs' metalinguistic abilities](https://mlsanity.com/paper/2305.00948/) (2023)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [Large language models and (non-)linguistic recursion](https://mlsanity.com/paper/2306.07195/) (2023)
- [LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs](https://mlsanity.com/paper/2507.16809/) (2025)
- [Sample-efficient Linguistic Generalizations through Program Synthesis: Experiments with Phonology Problems](https://mlsanity.com/paper/2106.06566/) (2021)
- [Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation Analysis](https://mlsanity.com/paper/2510.03366/) (2025)
- [Language Models Understand Us, Poorly](https://mlsanity.com/paper/2210.10684/) (2022)
- [UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?](https://mlsanity.com/paper/2508.11260/) (2025)
- [Can LLMs Generate and Solve Linguistic Olympiad Puzzles?](https://mlsanity.com/paper/2509.21820/) (2025)
- [Speaking of Language: Reflections on Metalanguage Research in NLP](https://mlsanity.com/paper/2604.02645/) (2026)
- [On Non-interactive Evaluation of Animal Communication Translators](https://mlsanity.com/paper/2510.15768/) (2025)
- [Can LLM Agents Identify Spoken Dialects like a Linguist?](https://mlsanity.com/paper/2603.29541/) (2026)
- [From Rosetta to Match-Up: A Paired Corpus of Linguistic Puzzles with Human and LLM Benchmarks](https://mlsanity.com/paper/2605.13408/) (2026)
- [SLATE: Are AI-Generated Slides Educationally Effective? A Benchmark for Language Teaching Quality and Learner Knowledge Acquisition](https://mlsanity.com/paper/2609.06212/) (2026)
- [LLM Agents as Computational Typologists](https://mlsanity.com/paper/2609.07791/) (2026)

## Abstract (arXiv)

Deep neural models have repeatedly proved excellent at memorizing surface patterns from large datasets for various ML and NLP benchmarks. They struggle to achieve human-like thinking, however, because they lack the skill of iterative reasoning upon knowledge. To expose this problem in a new light, we introduce a challenge on learning from small data, PuzzLing Machines, which consists of Rosetta Stone puzzles from Linguistic Olympiads for high school students. These puzzles are carefully designed to contain only the minimal amount of parallel text necessary to deduce the form of unseen expressions. Solving them does not require external information (e.g., knowledge bases, visual signals) or linguistic expertise, but meta-linguistic awareness and deductive skills. Our challenge contains around 100 puzzles covering a wide range of linguistic phenomena from 81 languages. We show that both simple statistical algorithms and state-of-the-art deep neural models perform inadequately on this challenge, as expected. We hope that this benchmark, available at https://ukplab.github.io/PuzzLing-Machines/, inspires further efforts towards a new paradigm in NLP---one that is grounded in human-like reasoning and understanding.