# Language, Language Models, and What We're Talking About

arXiv: https://arxiv.org/abs/2609.03577 · HTML: https://mlsanity.com/paper/2609.03577/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2504.08165/) (2025)
- [A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications](https://mlsanity.com/paper/2503.17003/) (2025)
- [The Shrinking Landscape of Linguistic Diversity in the Age of Large Language Models](https://mlsanity.com/paper/2502.11266/) (2025)
- [Fair is Better than Sensational:Man is to Doctor as Woman is to Doctor](https://mlsanity.com/paper/1905.09866/) (2019)
- [Large Language Models are Biased Because They Are Large Language Models](https://mlsanity.com/paper/2406.13138/) (2024)
- [IT5: Text-to-text Pretraining for Italian Language Understanding and Generation](https://mlsanity.com/paper/2203.03759/) (2022)
- [Advanced Natural-based interaction for the ITAlian language: LLaMAntino-3-ANITA](https://mlsanity.com/paper/2405.07101/) (2024)
- [VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models](https://mlsanity.com/paper/2410.12851/) (2024)
- [GePpeTto Carves Italian into a Language Model](https://mlsanity.com/paper/2004.14253/) (2020)
- [MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs](https://mlsanity.com/paper/2507.17476/) (2025)
- [Camoscio: an Italian Instruction-tuned LLaMA](https://mlsanity.com/paper/2307.16456/) (2023)
- [Alignment Makes Language Models Normative, Not Descriptive](https://mlsanity.com/paper/2603.17218/) (2026)
- [A gentle push funziona benissimo: making instructed models in Italian via contrastive activation steering](https://mlsanity.com/paper/2411.18247/) (2024)
- [GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-training](https://mlsanity.com/paper/2604.00920/) (2026)

## Abstract (arXiv)

Language models are commonly discussed as technical artefacts, but they are obviously shaped by the linguistic worlds conveyed by data during their training. Using Italian language models as evidence, I want to bring attention to the nature of the systems which result from training and specialising models on translated and synthetic data, and further curating them, and to the meaning of testing them on equally unnatural data. Are these eventually models of Italian? Are they models of language? Does NLP still care about language? These questions yield another, more concrete question: what language do we actually want language models to produce? I argue that this question cannot be answered if we do not first consider a clearer distinction between language models designed as technical products and language models designed as tools for studying language itself. The answers then might be diverse, the languages we are talking about might be diverse, and the picture might not be as pessimistic as we fear.