# Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research

arXiv: https://arxiv.org/abs/1908.10747 · HTML: https://mlsanity.com/paper/1908.10747/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2019.

## Built on by (most cited first, 6 of 6)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [Auxiliary task demands mask the capabilities of smaller language models](https://mlsanity.com/paper/2404.02418/) (2024)
- [Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategy](https://mlsanity.com/paper/2304.07007/) (2023)
- [A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench](https://mlsanity.com/paper/2507.08491/) (2025)
- [Playpen: An Environment for Exploring Learning Through Conversational Interaction](https://mlsanity.com/paper/2504.08590/) (2025)

## Abstract (arXiv)

"This paper introduces a new task and a new dataset", "we improve the state of the art in X by Y" -- it is rare to find a current natural language processing paper (or AI paper more generally) that does not contain such statements. What is mostly left implicit, however, is the assumption that this necessarily constitutes progress, and what it constitutes progress towards. Here, we make more precise the normally impressionistically used notions of language task and language game and ask how a research programme built on these might make progress towards the goal of modelling general language competence.