# Large Linguistic Models: Investigating LLMs' metalinguistic abilities

arXiv: https://arxiv.org/abs/2305.00948 · HTML: https://mlsanity.com/paper/2305.00948/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 83 references that are in the map.
- First posted on arXiv 1 May 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [What do RNN Language Models Learn about Filler-Gap Dependencies?](https://mlsanity.com/paper/1809.00042/) (2018)
- [Characterizing Intrinsic Compositionality in Transformers with Tree Projections](https://mlsanity.com/paper/2211.01288/) (2022)
- [Prompting Language Models for Linguistic Structure](https://mlsanity.com/paper/2211.07830/) (2022)
- [How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech](https://mlsanity.com/paper/2301.11462/) (2023)
- [PuzzLing Machines: A Challenge on Learning From Small Data](https://mlsanity.com/paper/2004.13161/) (2020)
- [CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networks](https://mlsanity.com/paper/2006.02951/) (2020)
- [ELQA: A Corpus of Metalinguistic Questions and Answers about English](https://mlsanity.com/paper/2205.00395/) (2022)
- [Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data](https://mlsanity.com/paper/2203.11476/) (2022)
- [Approaching an unknown communication system by latent space exploration and causal inference](https://mlsanity.com/paper/2303.10931/) (2023)

## Built on by (most cited first, 18 of 18)

- [Multi-Step Reasoning with Large Language Models, a Survey](https://mlsanity.com/paper/2407.11511/) (2024)
- [SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models](https://mlsanity.com/paper/2511.23075/) (2025)
- [Language Models as Models of Language](https://mlsanity.com/paper/2408.07144/) (2024)
- [ConlangCrafter: Constructing Languages with a Multi-Hop LLM Pipeline](https://mlsanity.com/paper/2508.06094/) (2025)
- [Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis](https://mlsanity.com/paper/2510.25356/) (2025)
- [Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean](https://mlsanity.com/paper/2506.01237/) (2025)
- [Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzles](https://mlsanity.com/paper/2506.13886/) (2025)
- [Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis](https://mlsanity.com/paper/2512.00214/) (2025)
- [IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation](https://mlsanity.com/paper/2604.04704/) (2026)
- [Statistical Parsing for Logical Information Retrieval](https://mlsanity.com/paper/2602.12170/) (2026)
- [Speaking of Language: Reflections on Metalanguage Research in NLP](https://mlsanity.com/paper/2604.02645/) (2026)
- [Metacognition in LLMs: Foundations, Progress, and Opportunities](https://mlsanity.com/paper/2607.11881/) (2026)
- [Grammar Engineering Meets LLMs: Development of Cantonese and Irish ParGram Treebanks](https://mlsanity.com/paper/2608.07283/) (2026)
- [GlossoGen: Emergent Language in Complex Multi-Agent LLM Interactions](https://mlsanity.com/paper/2609.01491/) (2026)
- [Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages](https://mlsanity.com/paper/2602.02182/) (2026)
- [Understanding LLM Failures: A Multi-Tape Turing Machine Analysis of Systematic Errors in Language Model Reasoning](https://mlsanity.com/paper/2602.15868/) (2026)
- [LLM Agents as Computational Typologists](https://mlsanity.com/paper/2609.07791/) (2026)
- [Co-Linguistics: AI-augmented Theory Construction in Linguistics](https://mlsanity.com/paper/2609.37635/) (2026)

## Abstract (arXiv)

The performance of large language models (LLMs) has recently improved to the point where models can perform well on many language tasks. We show here that--for the first time--the models can also generate valid metalinguistic analyses of language data. We outline a research program where the behavioral interpretability of LLMs on these tasks is tested via prompting. LLMs are trained primarily on text--as such, evaluating their metalinguistic abilities improves our understanding of their general capabilities and sheds new light on theoretical models in linguistics. We show that OpenAI's (2024) o1 vastly outperforms other models on tasks involving drawing syntactic trees and phonological generalization. We speculate that OpenAI o1's unique advantage over other models may result from the model's chain-of-thought mechanism, which mimics the structure of human reasoning used in complex cognitive tasks, such as linguistic analysis.