# On Reality and the Limits of Language Data: Aligning LLMs with Human Norms

arXiv: https://arxiv.org/abs/2208.11981 · HTML: https://mlsanity.com/paper/2208.11981/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 66 references that are in the map.
- First posted on arXiv 25 Aug 2022.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [CommonsenseQA 2.0: Exposing the Limits of AI through Gamification](https://mlsanity.com/paper/2201.05320/) (2022)
- [Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning](https://mlsanity.com/paper/2107.02794/) (2021)
- [Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches](https://mlsanity.com/paper/1904.01172/) (2019)
- [Do Neural Language Representations Learn Physical Commonsense?](https://mlsanity.com/paper/1908.02899/) (2019)
- [PROST: Physical Reasoning of Objects through Space and Time](https://mlsanity.com/paper/2106.03634/) (2021)
- [A Surprisingly Robust Trick for Winograd Schema Challenge](https://mlsanity.com/paper/1905.06290/) (2019)
- [A Review of Winograd Schema Challenge Datasets and Approaches](https://mlsanity.com/paper/2004.13831/) (2020)
- [Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations](https://mlsanity.com/paper/1910.01157/) (2019)
- [How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG](https://mlsanity.com/paper/1811.01778/) (2018)
- [Toward a New Science of Common Sense](https://mlsanity.com/paper/2112.12754/) (2021)

## Built on by (most cited first, 2 of 2)

- [ARN: Analogical Reasoning on Narratives](https://mlsanity.com/paper/2310.00996/) (2023)
- [Cognitive Science-Inspired Evaluation of Core Capabilities for Object Understanding in AI](https://mlsanity.com/paper/2503.21668/) (2025)

## Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) harness linguistic associations in vast natural language data for practical applications. However, their ability to understand the physical world using only language data remains a question. After reviewing existing protocols, we explore this question using a novel and tightly controlled reasoning test (ART) and compare human norms against versions of GPT-3. Our findings highlight the categories of common-sense relations models that could learn directly from data and areas of weakness. GPT-3 offers evidence for verbal reasoning on a par with human subjects for several relations including Synonymy, Antonymy, and Default inheritance, Without reinforcement learning from human judgements, it appears GPT-3 performs at the lower end of the reference interval for Has-part and Contained-in. Weaknesses were observed also in affordance characteristics through Necessary-quality, Order-of-size and Order-of-intensity. Combining LLMs with symbolic world grounding is a promising direction to address associative learning.