# MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs

arXiv: https://arxiv.org/abs/2609.35312 · HTML: https://mlsanity.com/paper/2609.35312/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 13 of its 37 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 13 of 13)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)
- [Olmo 3](https://mlsanity.com/paper/2512.13961/) (2025)
- [Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap](https://mlsanity.com/paper/2402.19450/) (2024)
- [VERINA: Benchmarking Verifiable Code Generation](https://mlsanity.com/paper/2505.23135/) (2025)
- [Cofca: A Step-Wise Counterfactual Multi-hop QA benchmark](https://mlsanity.com/paper/2402.11924/) (2024)
- [RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content](https://mlsanity.com/paper/2406.11811/) (2024)
- [Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges](https://mlsanity.com/paper/2502.08680/) (2025)
- [Beyond Consensus: Mitigating the Agreeableness Bias in LLM Judge Evaluations](https://mlsanity.com/paper/2510.11822/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) perform well on reasoning benchmarks, but it remains unclear whether this reflects genuine contextual reasoning or reliance on facts memorized in their parameters. We investigate this by distinguishing two possibilities: a broad textitmemorization bias, where familiar content improves reasoning performance, and the textitStrong Parametric Shortcut Hypothesis, where models skip reasoning entirely and recall stored answers. To test these effects, we introduce MemoReason, a human-curated benchmark that pairs factual reasoning tasks with structurally identical fictitiousterm versions where real entities like people, companies, or dates are systematically replaced by fictitiousterm ones of the same type. This scorerevisionpreserves task structure and specified reasoning operations while varying the familiarity of the context, allowing controlled measurement of how the parametric memory affects reasoning. revisionOur evaluation of recent LLMs reveals consistent and statistically significant performance drops of up to 15.7% in the fictitious setting, demonstrating a clear memorization bias. However, a targeted analysis of revisionquestions failed in the fictitious setting shows that models rarely respond with the corresponding factual answer, indicating that direct parametric shortcuts are not the dominant failure mode. These findings suggest that parametric memory influences reasoning through mechanisms more complex than simple factual recall. MemoReason provides a controlled framework for studying these mechanisms and for extending paired factual-fictitious evaluation to broader reasoning settings.