# Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?

arXiv: https://arxiv.org/abs/2104.10809 · HTML: https://mlsanity.com/paper/2104.10809/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 36 references that are in the map.
- First posted on arXiv 22 Apr 2021.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [Learning and Evaluating General Linguistic Intelligence](https://mlsanity.com/paper/1901.11373/) (2019)
- [WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge](https://mlsanity.com/paper/2005.05763/) (2020)

## Built on by (most cited first, 30 of 55)

- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color](https://mlsanity.com/paper/2109.06129/) (2021)
- [Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models](https://mlsanity.com/paper/2404.03622/) (2024)
- [Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions](https://mlsanity.com/paper/2212.10561/) (2022)
- [A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task](https://mlsanity.com/paper/2402.11917/) (2024)
- [Entity Tracking in Language Models](https://mlsanity.com/paper/2305.02363/) (2023)
- [Contrastive language and vision learning of general fashion concepts](https://mlsanity.com/paper/2204.03972/) (2022)
- [The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail](https://mlsanity.com/paper/2110.08300/) (2021)
- [Emergent Representations of Program Semantics in Language Models Trained on Programs](https://mlsanity.com/paper/2305.11169/) (2023)
- [What Do NLP Researchers Believe? Results of the NLP Community Metasurvey](https://mlsanity.com/paper/2208.12852/) (2022)
- [Does Vision-and-Language Pretraining Improve Lexical Grounding?](https://mlsanity.com/paper/2109.10246/) (2021)
- [When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it](https://mlsanity.com/paper/2205.03472/) (2022)
- [Unit Testing for Concepts in Neural Networks](https://mlsanity.com/paper/2208.10244/) (2022)
- [In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax](https://mlsanity.com/paper/2311.07811/) (2023)
- [On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations](https://mlsanity.com/paper/2204.12708/) (2022)
- [KILM: Knowledge Injection into Encoder-Decoder Language Models](https://mlsanity.com/paper/2302.09170/) (2023)
- [Meaning Representations from Trajectories in Autoregressive Models](https://mlsanity.com/paper/2310.18348/) (2023)
- [LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond](https://mlsanity.com/paper/2105.12449/) (2021)
- [ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?](https://mlsanity.com/paper/2403.17368/) (2024)
- [What Do Language Models Hear? Probing for Auditory Representations in Language Models](https://mlsanity.com/paper/2402.16998/) (2024)
- [Emergent Language: A Survey and Taxonomy](https://mlsanity.com/paper/2409.02645/) (2024)
- [Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling](https://mlsanity.com/paper/2112.07874/) (2021)
- [Entailment Semantics Can Be Extracted from an Ideal Language Model](https://mlsanity.com/paper/2209.12407/) (2022)
- [Can You Learn Semantics Through Next-Word Prediction? The Case of Entailment](https://mlsanity.com/paper/2402.13956/) (2024)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge](https://mlsanity.com/paper/2112.08583/) (2021)
- [T5QL: Taming language models for SQL generation](https://mlsanity.com/paper/2209.10254/) (2022)

## Abstract (arXiv)

Language models trained on billions of tokens have recently led to unprecedented results on many NLP tasks. This success raises the question of whether, in principle, a system can ever ``understand'' raw text without access to some form of grounding. We formally investigate the abilities of ungrounded systems to acquire meaning. Our analysis focuses on the role of ``assertions'': textual contexts that provide indirect clues about the underlying semantics. We study whether assertions enable a system to emulate representations preserving semantic relations like equivalence. We find that assertions enable semantic emulation of languages that satisfy a strong notion of semantic transparency. However, for classes of languages where the same expression can take different values in different contexts, we show that emulation can become uncomputable. Finally, we discuss differences between our formal model and natural language, exploring how our results generalize to a modal setting and other semantic relations. Together, our results suggest that assertions in code or language do not provide sufficient signal to fully emulate semantic representations. We formalize ways in which ungrounded language models appear to be fundamentally limited in their ability to ``understand''.