# Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models

arXiv: https://arxiv.org/abs/2405.09605 · HTML: https://mlsanity.com/paper/2405.09605/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 48 of its 79 references that are in the map.
- First posted on arXiv 15 May 2024.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 30 of 48)

- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Evaluating Large Language Models in Theory of Mind Tasks](https://mlsanity.com/paper/2302.02083/) (2023)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)

## Built on by (most cited first, 30 of 55)

- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory](https://mlsanity.com/paper/2508.09736/) (2025)
- [MuMA-ToM: Multi-modal Multi-Agent Theory of Mind](https://mlsanity.com/paper/2408.12574/) (2024)
- [GPT or BERT: why not both?](https://mlsanity.com/paper/2410.24159/) (2024)
- [From Insights to Actions: The Impact of Interpretability and Analysis Research on NLP](https://mlsanity.com/paper/2406.12618/) (2024)
- [Potemkin Understanding in Large Language Models](https://mlsanity.com/paper/2506.21521/) (2025)
- [Once Upon a Time: Interactive Learning for Storytelling with Small Language Models](https://mlsanity.com/paper/2509.15714/) (2025)
- [Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)](https://mlsanity.com/paper/2510.20358/) (2025)
- [The Future of Open Human Feedback](https://mlsanity.com/paper/2408.16961/) (2024)
- [BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data](https://mlsanity.com/paper/2510.10159/) (2025)
- [Mixture of Cognitive Reasoners: Modular Reasoning with Brain-Like Specialization](https://mlsanity.com/paper/2506.13331/) (2025)
- [Pretraining Language Models for Diachronic Linguistic Change Discovery](https://mlsanity.com/paper/2504.05523/) (2025)
- [BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data](https://mlsanity.com/paper/2409.17312/) (2024)
- [What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performance](https://mlsanity.com/paper/2411.06672/) (2024)
- [BabyLM Challenge: Exploring the Effect of Variation Sets on Language Model Training Efficiency](https://mlsanity.com/paper/2411.09587/) (2024)
- [Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaning](https://mlsanity.com/paper/2506.04408/) (2025)
- [BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training](https://mlsanity.com/paper/2410.15365/) (2024)
- [Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Training](https://mlsanity.com/paper/2410.15509/) (2024)
- [Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Data](https://mlsanity.com/paper/2411.00828/) (2024)
- [Trick or Neat: Adversarial Ambiguity and Language Model Evaluation](https://mlsanity.com/paper/2506.01205/) (2025)
- [Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility](https://mlsanity.com/paper/2507.12553/) (2025)
- [Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning](https://mlsanity.com/paper/2507.19586/) (2025)
- [LongTail-Swap: benchmarking language models' abilities on rare words](https://mlsanity.com/paper/2510.04268/) (2025)
- [Revisiting Rogers' Paradox in the Context of Human-AI Interaction](https://mlsanity.com/paper/2501.10476/) (2025)
- [The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories](https://mlsanity.com/paper/2501.12651/) (2025)
- [Influence-driven Curriculum Learning for Pre-training on Limited Data](https://mlsanity.com/paper/2508.15475/) (2025)
- [Masked Diffusion Language Models with Frequency-Informed Training](https://mlsanity.com/paper/2509.05056/) (2025)
- [Rethinking the Role of Text Complexity in Language Model Pretraining](https://mlsanity.com/paper/2509.16551/) (2025)
- [WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectives](https://mlsanity.com/paper/2510.09556/) (2025)
- [Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models](https://mlsanity.com/paper/2505.04914/) (2025)

## Abstract (arXiv)

The ability to build and reason about models of the world is essential for situated language understanding. But evaluating world modeling capabilities in modern AI systems -- especially those based on language models -- has proven challenging, in large part because of the difficulty of disentangling conceptual knowledge about the world from knowledge of surface co-occurrence statistics. This paper presents Elements of World Knowledge (EWoK), a framework for evaluating language models' understanding of the conceptual knowledge underlying world modeling. EWoK targets specific concepts from multiple knowledge domains known to be important for world modeling in humans, from social interactions (help, deceive) to spatial relations (left, right). Objects, agents, and locations in the items can be flexibly filled in, enabling easy generation of multiple controlled datasets. We then introduce EWoK-core-1.0, a dataset of 4,374 items covering 11 world knowledge domains. We evaluate 20 open-weights large language models (1.3B--70B parameters) and compare them with human performance. All tested models perform worse than humans, with results varying drastically across domains. Performance on social interactions and social properties was highest and performance on physical relations and spatial relations was lowest. Overall, this dataset highlights simple cases where even large models struggle and presents rich avenues for targeted research on LLM world modeling capabilities.