# Chatbots Output Meaningful (but Problematic) Language

arXiv: https://arxiv.org/abs/2606.02973 · HTML: https://mlsanity.com/paper/2606.02973/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 91 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG](https://mlsanity.com/paper/2501.09136/) (2025)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions](https://mlsanity.com/paper/2310.15421/) (2023)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [Sycophancy in Large Language Models: Causes and Mitigations](https://mlsanity.com/paper/2411.15287/) (2024)
- [Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses](https://mlsanity.com/paper/2104.14694/) (2021)
- [Do Language Models' Words Refer?](https://mlsanity.com/paper/2308.05576/) (2023)
- [Investigating Memorization of Conspiracy Theories in Text Generation](https://mlsanity.com/paper/2101.00379/) (2021)
- [Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective](https://mlsanity.com/paper/2504.10839/) (2025)
- [Are Language Models More Like Libraries or Like Librarians? Bibliotechnism, the Novel Reference Problem, and the Attitudes of LLMs](https://mlsanity.com/paper/2401.04854/) (2024)
- [Generation Challenges: Results of the Accuracy Evaluation Shared Task](https://mlsanity.com/paper/2108.05644/) (2021)
- [Recognizing Artificial Minds: A Philosophical Defense of AI Cognition](https://mlsanity.com/paper/2504.13988/) (2025)
- [The Outputs of Large Language Models are Meaningless](https://mlsanity.com/paper/2509.22206/) (2025)

## Abstract (arXiv)

Are utterances by AI chatbots meaningful? Concretely, if a user asks, say, Anthropic's agent Claude, "What is the capital of Spain?" and Claude answers, "Madrid is the capital of Spain," does that sentence have its ordinary meaning -- and does it express a true proposition? Most ordinary users, as well as AI engineers, take the answer to be trivially "yes." However, many cognitive scientists, linguists, and philosophers of language argue that dominant intentionalist accounts of language and meaning deliver the opposite conclusion. Theorists more sympathetic to ordinary users' intuitions have therefore advocated a radical "de-anthropomorphization" of language, revising our understanding of mental states, intentions, and semantic content to capture the intuition that the outputs of LLMs are meaningful. We take a different approach. While we, too, argue that LLM outputs are meaningful, we contend that a proper theory of human language already applies, as is, to current chatbots. Meaning is a low bar: claiming that LLM outputs are meaningful does not require positing mental states, intentions, rationality, or the cognitive capacities requisite for communication in LLMs -- or, indeed, making any other anthropomorphic assumptions. People do have communicative intentions (typically successful ones), but nevertheless, even in humans, language production can depart from what the speaker has in mind. Our view has important consequences for how we should theorize about -- and critically engage with -- both human linguistic output and synthetically generated text. In particular, to say that chatbots produce meaningful text is not by any means to endorse what they output, or to assume that the technology is (or is not) good, powerful, appropriate, or useful.