# ConlangBench: Exploring Language Knowledge and Learning in LLMs through Diverse Constructed Languages

arXiv: https://arxiv.org/abs/2608.03505 · HTML: https://mlsanity.com/paper/2608.03505/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 53 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Builds on (most cited first, 14 of 14)

- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens](https://mlsanity.com/paper/2401.17377/) (2024)
- [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020)
- [A Benchmark for Learning to Translate a New Language from One Grammar Book](https://mlsanity.com/paper/2309.16575/) (2023)
- [The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities](https://mlsanity.com/paper/2411.04986/) (2024)
- [Emergent Word Order Universals from Cognitively-Motivated Language Models](https://mlsanity.com/paper/2402.12363/) (2024)
- [Searching for Structure: Investigating Emergent Communication with Large Language Models](https://mlsanity.com/paper/2412.07646/) (2024)
- [ConlangCrafter: Constructing Languages with a Multi-Hop LLM Pipeline](https://mlsanity.com/paper/2508.06094/) (2025)
- [The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang](https://mlsanity.com/paper/2509.00425/) (2025)
- [Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism](https://mlsanity.com/paper/2511.10045/) (2025)
- [A Network Analysis Approach to Conlang Research Literature](https://mlsanity.com/paper/2407.15370/) (2024)
- [Implicit In-Context Learning: Evidence from Artificial Language Experiments](https://mlsanity.com/paper/2503.24190/) (2025)
- [Creating ConLangs to Probe the Metalinguistic Grammatical Knowledge of LLMs](https://mlsanity.com/paper/2510.07591/) (2025)
- [Talking with Oompa Loompas: A novel framework for evaluating linguistic acquisition of LLM agents](https://mlsanity.com/paper/2509.07389/) (2025)

## Abstract (arXiv)

Constructed languages (conlangs) are intentionally created human languages with a rich tradition of linguistic creativity. Despite their potential for studying language learning in large language models (LLMs), existing conlangs remain largely underexplored in LLM research. We present ConlangBench, the first large-scale benchmark for evaluating and training LLMs on 21 existing conlangs. We collect over 21M conlang-English parallel sentence pairs (including 430K pairs across the 20 non-Esperanto conlangs) and 321K vocabulary entries. In bidirectional translation experiments, we find that models perform better on a posteriori conlangs, whose vocabularies are derived from natural languages, reflecting the design characteristics of conlangs. Training on ConlangBench also shows that models can learn all eight conlangs for which sufficient parallel corpora are available, while their learning curves vary depending on how the conlangs were created. Our findings suggest that conlangs provide a unique testbed for investigating how LLMs acquire low-resource languages.