# Memorization neq Understanding: Do Large Language Models Have the Ability of Scenario Cognition?

arXiv: https://arxiv.org/abs/2509.04866 · HTML: https://mlsanity.com/paper/2509.04866/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 5 Sep 2025.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024)
- [Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data](https://mlsanity.com/paper/2407.14985/) (2024)
- [Undesirable Memorization in Large Language Models: A Survey](https://mlsanity.com/paper/2410.02650/) (2024)
- [Atomic Fact Decomposition Helps Attributed Question Answering](https://mlsanity.com/paper/2410.16708/) (2024)
- [Explaining Black-box Language Models with Knowledge Probing Systems: A Post-hoc Explanation Perspective](https://mlsanity.com/paper/2508.16969/) (2025)

## Built on by (most cited first, 1 of 1)

- [Uncovering and Mitigating Transient Blindness in Multimodal Model Editing](https://mlsanity.com/paper/2511.13243/) (2025)

## Abstract (arXiv)

Driven by vast and diverse textual data, large language models (LLMs) have demonstrated impressive performance across numerous natural language processing (NLP) tasks. Yet, a critical question persists: does their generalization arise from mere memorization of training data or from deep semantic understanding? To investigate this, we propose a bi-perspective evaluation framework to assess LLMs' scenario cognition - the ability to link semantic scenario elements with their arguments in context. Specifically, we introduce a novel scenario-based dataset comprising diverse textual descriptions of fictional facts, annotated with scenario elements. LLMs are evaluated through their capacity to answer scenario-related questions (model output perspective) and via probing their internal representations for encoded scenario elements-argument associations (internal representation perspective). Our experiments reveal that current LLMs predominantly rely on superficial memorization, failing to achieve robust semantic scenario cognition, even in simple cases. These findings expose critical limitations in LLMs' semantic understanding and offer cognitive insights for advancing their capabilities.