mlsanityData to 7 Oct 2026

Large Linguistic Models: Investigating LLMs' metalinguistic abilities

arXiv 2305.00948 ↗First posted 1 May 2023cs.CL cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

The performance of large language models (LLMs) has recently improved to the point where models can perform well on many language tasks. We show here that--for the first time--the models can also generate valid metalinguistic analyses of language data. We outline a research program where the behavioral interpretability of LLMs on these tasks is tested via prompting. LLMs are trained primarily on text--as such, evaluating their metalinguistic abilities improves our understanding of their general capabilities and sheds new light on theoretical models in linguistics. We show that OpenAI's (2024) o1 vastly outperforms other models on tasks involving drawing syntactic trees and phonological generalization. We speculate that OpenAI o1's unique advantage over other models may result from the model's chain-of-thought mechanism, which mimics the structure of human reasoning used in complex cognitive tasks, such as linguistic analysis.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 15 of its 83 references that are in the map. First posted on arXiv 1 May 2023.

Builds on

15 of 83 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  5. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  6. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  7. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  8. 2022Characterizing Intrinsic Compositionality in Transformers with Tree Projectionscited by 40
  9. 2022Prompting Language Models for Linguistic Structurecited by 39
  10. 2023How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speechcited by 22
  11. 2020PuzzLing Machines: A Challenge on Learning From Small Datacited by 15
  12. 2020CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networkscited by 15
3 more
  1. 2022ELQA: A Corpus of Metalinguistic Questions and Answers about Englishcited by 8
  2. 2022Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech datacited by 5
  3. 2023Approaching an unknown communication system by latent space exploration and causal inferencecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 10.

18 papers.

Sort:

By area:

  1. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  2. 2024Language Models as Models of Languagecited by 24
  3. 2025SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Modelscited by 21
  4. 2025ConlangCrafter: Constructing Languages with a Multi-Hop LLM Pipelinecited by 4
  5. 2025Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysiscited by 4
  6. 2025Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Koreancited by 2
  7. 2025Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzlescited by 2
  8. 2025Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysiscited by 2
  9. 2026Statistical Parsing for Logical Information Retrievalcited by 1
  10. 2026Speaking of Language: Reflections on Metalanguage Research in NLPcited by 1
  11. 2026IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variationcited by 1
  12. 2026Metacognition in LLMs: Foundations, Progress, and Opportunitiescited by 1
6 more
  1. 2026Grammar Engineering Meets LLMs: Development of Cantonese and Irish ParGram Treebankscited by 1
  2. 2026GlossoGen: Emergent Language in Complex Multi-Agent LLM Interactionscited by 1
  3. 2026Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languagescited by 0
  4. 2026Understanding LLM Failures: A Multi-Tape Turing Machine Analysis of Systematic Errors in Language Model Reasoningcited by 0
  5. 2026LLM Agents as Computational Typologistscited by 0
  6. 2026Co-Linguistics: AI-augmented Theory Construction in Linguisticscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang
  2. 2026Understanding Large Language Models
  3. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  4. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  5. 2024When a language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAI o1
  6. 2024LLMs' Understanding of Natural Language Revealed
  7. 2025IOLBENCH: Benchmarking LLMs on Linguistic Reasoning
  8. 2025A Survey on Large Language Models with some Insights on their Capabilities and Limitations