# I am a Strange Dataset: Metalinguistic Tests for Language Models

arXiv: https://arxiv.org/abs/2401.05300 · HTML: https://mlsanity.com/paper/2401.05300/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 10 Jan 2024.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://mlsanity.com/paper/2308.07921/) (2023)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation](https://mlsanity.com/paper/2310.02304/) (2023)
- [Fine-Tuning Language Models with Advantage-Induced Policy Alignment](https://mlsanity.com/paper/2306.02231/) (2023)
- [Verb Argument Structure Alternations in Word and Sentence Embeddings](https://mlsanity.com/paper/1811.10773/) (2018)
- [Investigating Novel Verb Learning in BERT: Selectional Preference Classes and Alternation-Based Syntactic Generalization](https://mlsanity.com/paper/2011.02417/) (2020)
- [ELQA: A Corpus of Metalinguistic Questions and Answers about English](https://mlsanity.com/paper/2205.00395/) (2022)
- [CuRIAM: Corpus re Interpretation and Metalanguage in U.S. Supreme Court Opinions](https://mlsanity.com/paper/2305.14719/) (2023)

## Built on by (most cited first, 10 of 10)

- [LLMs Will Always Hallucinate, and We Need to Live With This](https://mlsanity.com/paper/2409.05746/) (2024)
- [Language Models Fail to Introspect About Their Knowledge of Language](https://mlsanity.com/paper/2503.07513/) (2025)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [Do language models practice what they preach? Examining language ideologies about gendered language reform encoded in LLMs](https://mlsanity.com/paper/2409.13852/) (2024)
- [Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis](https://mlsanity.com/paper/2510.25356/) (2025)
- [Analogical Structure, Minimal Contextual Cues and Contrastive Distractors: Input Design for Sample-Efficient Linguistic Rule Induction](https://mlsanity.com/paper/2511.10441/) (2025)
- [Speaking of Language: Reflections on Metalanguage Research in NLP](https://mlsanity.com/paper/2604.02645/) (2026)
- [Phantom Paradoxes: Language Models Treat Unresolved Circular Truth Dependence as Contradiction](https://mlsanity.com/paper/2604.12128/) (2026)
- [Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution Autonomous Driving VQA from Peru](https://mlsanity.com/paper/2503.07587/) (2025)
- [Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages](https://mlsanity.com/paper/2602.02182/) (2026)

## Abstract (arXiv)

Statements involving metalinguistic self-reference ("This paper has six sections.") are prevalent in many domains. Can current large language models (LLMs) handle such language? In this paper, we present "I am a Strange Dataset", a new dataset for addressing this question. There are two subtasks: generation and verification. In generation, models continue statements like "The penultimate word in this sentence is" (where a correct continuation is "is"). In verification, models judge the truth of statements like "The penultimate word in this sentence is sentence." (false). We also provide minimally different metalinguistic non-self-reference examples to complement the main dataset by probing for whether models can handle metalinguistic language at all. The dataset is hand-crafted by experts and validated by non-expert annotators. We test a variety of open-source LLMs (7B to 70B parameters) as well as closed-source LLMs through APIs. All models perform close to chance across both subtasks and even on the non-self-referential metalinguistic control data, though we find some steady improvement with model scale. GPT 4 is the only model to consistently do significantly better than chance, and it is still only in the 60% range, while our untrained human annotators score well in the 89-93% range. The dataset and evaluation toolkit are available at https://github.com/TristanThrush/i-am-a-strange-dataset.