mlsanityData to 7 Oct 2026

I am a Strange Dataset: Metalinguistic Tests for Language Models

arXiv 2401.05300 ↗First posted 10 Jan 2024cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Statements involving metalinguistic self-reference ("This paper has six sections.") are prevalent in many domains. Can current large language models (LLMs) handle such language? In this paper, we present "I am a Strange Dataset", a new dataset for addressing this question. There are two subtasks: generation and verification. In generation, models continue statements like "The penultimate word in this sentence is" (where a correct continuation is "is"). In verification, models judge the truth of statements like "The penultimate word in this sentence is sentence." (false). We also provide minimally different metalinguistic non-self-reference examples to complement the main dataset by probing for whether models can handle metalinguistic language at all. The dataset is hand-crafted by experts and validated by non-expert annotators. We test a variety of open-source LLMs (7B to 70B parameters) as well as closed-source LLMs through APIs. All models perform close to chance across both subtasks and even on the non-self-referential metalinguistic control data, though we find some steady improvement with model scale. GPT 4 is the only model to consistently do significantly better than chance, and it is still only in the 60% range, while our untrained human annotators score well in the 89-93% range. The dataset and evaluation toolkit are available at https://github.com/TristanThrush/i-am-a-strange-dataset.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 19 of its 45 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

19 of 45 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  4. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  6. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  7. 2018Gender Bias in Coreference Resolutioncited by 438
  8. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  9. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  10. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  11. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  12. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
7 more
  1. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  2. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  3. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  4. 2018Verb Argument Structure Alternations in Word and Sentence Embeddingscited by 25
  5. 2020Investigating Novel Verb Learning in BERT: Selectional Preference Classes and Alternation-Based Syntactic Generalizationcited by 16
  6. 2022ELQA: A Corpus of Metalinguistic Questions and Answers about Englishcited by 8
  7. 2023CuRIAM: Corpus re Interpretation and Metalanguage in U.S. Supreme Court Opinionscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 3.

10 papers.

  1. 2024LLMs Will Always Hallucinate, and We Need to Live With Thiscited by 77
  2. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 28
  3. 2024Natural Language Processing RELIES on Linguisticscited by 9
  4. 2024Do language models practice what they preach? Examining language ideologies about gendered language reform encoded in LLMscited by 5
  5. 2025Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysiscited by 4
  6. 2025Analogical Structure, Minimal Contextual Cues and Contrastive Distractors: Input Design for Sample-Efficient Linguistic Rule Inductioncited by 1
  7. 2026Speaking of Language: Reflections on Metalanguage Research in NLPcited by 1
  8. 2026When Self-Reference Fails to Close: Matrix-Level Dynamics in Large Language Modelscited by 1
  9. 2025Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution Autonomous Driving VQA from Perucited by 0
  10. 2026Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languagescited by 0