# Speaking of Language: Reflections on Metalanguage Research in NLP

arXiv: https://arxiv.org/abs/2604.02645 · HTML: https://mlsanity.com/paper/2604.02645/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 83 references that are in the map.
- First posted on arXiv 3 Apr 2026.

## Builds on (most cited first, 30 of 31)

- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [A Benchmark for Learning to Translate a New Language from One Grammar Book](https://mlsanity.com/paper/2309.16575/) (2023)
- [Language Models Fail to Introspect About Their Knowledge of Language](https://mlsanity.com/paper/2503.07513/) (2025)
- [LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages](https://mlsanity.com/paper/2406.06196/) (2024)
- [Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?](https://mlsanity.com/paper/2409.19151/) (2024)
- [NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps](https://mlsanity.com/paper/2404.01651/) (2024)
- [Large Linguistic Models: Investigating LLMs' metalinguistic abilities](https://mlsanity.com/paper/2305.00948/) (2023)
- [Shortcomings of LLMs for Low-Resource Translation: Retrieval and Understanding are Both the Problem](https://mlsanity.com/paper/2406.15625/) (2024)
- [Linguini: A benchmark for language-agnostic linguistic reasoning](https://mlsanity.com/paper/2409.12126/) (2024)
- [Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP](https://mlsanity.com/paper/2104.08620/) (2021)
- [PuzzLing Machines: A Challenge on Learning From Small Data](https://mlsanity.com/paper/2004.13161/) (2020)
- ["You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation](https://mlsanity.com/paper/2310.17793/) (2023)
- [SIGTYP 2020 Shared Task: Prediction of Typological Features](https://mlsanity.com/paper/2010.08246/) (2020)
- [Understanding In-Context Machine Translation for Low-Resource Languages: A Case Study on Manchu](https://mlsanity.com/paper/2502.11862/) (2025)
- [Back to School: Translation Using Grammar Books](https://mlsanity.com/paper/2410.15263/) (2024)
- [Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling](https://mlsanity.com/paper/2112.07874/) (2021)
- [Lost in Translationese? Reducing Translation Effect Using Abstract Meaning Representation](https://mlsanity.com/paper/2304.11501/) (2023)
- [I am a Strange Dataset: Metalinguistic Tests for Language Models](https://mlsanity.com/paper/2401.05300/) (2024)
- [Automatic Extraction of Rules Governing Morphological Agreement](https://mlsanity.com/paper/2010.01160/) (2020)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [ELQA: A Corpus of Metalinguistic Questions and Answers about English](https://mlsanity.com/paper/2205.00395/) (2022)
- [The Past, Present, and Future of Typological Databases in NLP](https://mlsanity.com/paper/2310.13440/) (2023)
- [Explicit Learning and the LLM in Machine Translation](https://mlsanity.com/paper/2503.09454/) (2025)
- [LingGym: How Far Are LLMs from Thinking Like Field Linguists?](https://mlsanity.com/paper/2511.00343/) (2025)
- [When is Wall a Pared and when a Muro? -- Extracting Rules Governing Lexical Selection](https://mlsanity.com/paper/2109.06014/) (2021)
- [Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis](https://mlsanity.com/paper/2510.25356/) (2025)
- [Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning](https://mlsanity.com/paper/2206.05154/) (2022)

## Built on by (most cited first, 1 of 1)

- [LLM Agents as Computational Typologists](https://mlsanity.com/paper/2609.07791/) (2026)

## Abstract (arXiv)

This work aims to shine a spotlight on the topic of metalanguage. We first define metalanguage, link it to NLP and LLMs, and then discuss our two labs' metalanguage-centered efforts. Finally, we discuss four dimensions of metalanguage and metalinguistic tasks, offering a list of understudied future research directions.