# The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang

arXiv: https://arxiv.org/abs/2509.00425 · HTML: https://mlsanity.com/paper/2509.00425/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 12 of its 118 references that are in the map.
- First posted on arXiv 30 Aug 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 12 of 12)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition](https://mlsanity.com/paper/2310.05492/) (2023)
- [Examining the Inductive Bias of Neural Language Models with Artificial Languages](https://mlsanity.com/paper/2106.01044/) (2021)
- [Teaching Large Language Models an Unseen Language on the Fly](https://mlsanity.com/paper/2402.19167/) (2024)
- [A neural interlingua for multilingual machine translation](https://mlsanity.com/paper/1804.08198/) (2018)
- [Jump to better conclusions: SCAN both left and right](https://mlsanity.com/paper/1809.04640/) (2018)
- [See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses](https://mlsanity.com/paper/2408.08978/) (2024)
- [Back to School: Translation Using Grammar Books](https://mlsanity.com/paper/2410.15263/) (2024)
- [IOLBENCH: Benchmarking LLMs on Linguistic Reasoning](https://mlsanity.com/paper/2501.04249/) (2025)
- [modeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models](https://mlsanity.com/paper/2406.17038/) (2024)
- [LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs](https://mlsanity.com/paper/2507.16809/) (2025)
- [Explain-then-Process: Using Grammar Prompting to Enhance Grammatical Acceptability Judgments](https://mlsanity.com/paper/2506.02302/) (2025)

## Built on by (most cited first, 3 of 3)

- [Diagnosing CFG Interpretation in LLMs](https://mlsanity.com/paper/2604.20811/) (2026)
- [LLM Probe: Evaluating LLMs for Low-Resource Languages](https://mlsanity.com/paper/2603.29517/) (2026)
- [ConlangBench: Exploring Language Knowledge and Learning in LLMs through Diverse Constructed Languages](https://mlsanity.com/paper/2608.03505/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) achieve gold-medal performance across many benchmarks, yet it remains unclear whether such success reflects genuine reasoning or pattern matching. From a cognitive science perspective, an informative test is whether models can master an unfamiliar language through explicit metalinguistic deductive learning, a paradigm where human learners can reliably internalise grammatical systems through metalinguistic reasoning. We address this question with Camlang, a novel constructed language that exhibits naturalistic yet unattested feature combinations. Camlang consists of two explicit resources, a grammar book and a bilingual dictionary, which mirror adult second-language learning via explicit grammar rules and lexical lookup, and enable us to disentangle errors in morpho-syntax, lexical semantics, and sentence-level reasoning. Human experiments show that these resources are sufficient for participants to acquire Camlang and successfully solve Camlang tasks. To operationalise evaluation, we adapt CommonsenseQA into Camlang, creating Camlang-CSQA-v0, the first task in a broader suite where solving questions requires applying grammar rules and lexical mappings. Experimental results show that GPT-5 achieves 98% EM accuracy in English but only 47% in Camlang, far below human performance at 87%, while other state-of-the-art reasoning LLMs perform even worse. Human verification further reveals that most model successes stem from shallow lexical alignment while GPT-5 shows emerging metalinguistic awareness to a limited extent but not systematic grammatical mastery as humans. Camlang establishes a cognitively grounded evaluation paradigm that exposes fundamental gaps between current models and human metalinguistic competence.