# Beyond Memorization: Distinguishing Between Pattern-Based and Epistemic Reasoning in LLMs Using Epistemic Puzzles

arXiv: https://arxiv.org/abs/2603.21350 · HTML: https://mlsanity.com/paper/2603.21350/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 6 of its 23 references that are in the map.
- First posted on arXiv 22 Mar 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 6 of 6)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning](https://mlsanity.com/paper/2502.01100/) (2025)
- [Cooperative Epistemic Multi-Agent Planning for Implicit Coordination](https://mlsanity.com/paper/1703.02196/) (2017)
- [HardcoreLogic: Challenging Large Reasoning Models with Long-tail Logic Puzzle Games](https://mlsanity.com/paper/2510.12563/) (2025)

## Abstract (arXiv)

Epistemic reasoning requires agents to infer the state of the world from partial observations and information about other agents' knowledge. Prior work evaluating LLMs on epistemic puzzles often frames failures as memorization rather than reasoning. We argue that this dichotomy is too coarse for newer models: memorization is a limiting case of pattern-based reasoning, where a model matches a task to a familiar template and applies the corresponding solution. We introduce a two-dimensional benchmark over Dynamic Epistemic Logic (DEL) puzzles, separating narrative familiarity from inference complexity, which allows us to distinguish pattern-based from epistemic reasoning. We find that models are substantially more robust to surface form changes than prior work suggested, yet consistently struggle in asymmetric settings where familiar patterns no longer apply and success requires tracking fragmented epistemic states. Code, datasets, and evaluation results are available at https://github.com/adigabay84/del-puzzles-benchmark.