# EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges

arXiv: https://arxiv.org/abs/2502.08859 · HTML: https://mlsanity.com/paper/2502.08859/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 2 of its 31 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 2 of 2)

- [Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter?](https://mlsanity.com/paper/2407.14790/) (2024)
- [FCoReBench: Can Large Language Models Solve Challenging First-Order Combinatorial Reasoning Problems?](https://mlsanity.com/paper/2402.02611/) (2024)

## Built on by (most cited first, 19 of 19)

- [RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2504.20073/) (2025)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers](https://mlsanity.com/paper/2506.23918/) (2025)
- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2505.24760/) (2025)
- [lmgame-Bench: How Good are LLMs at Playing Games?](https://mlsanity.com/paper/2505.15146/) (2025)
- [VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2503.23064/) (2025)
- [Sudoku-Bench: Evaluating creative reasoning with Sudoku variants](https://mlsanity.com/paper/2505.16135/) (2025)
- [General Modular Harness for LLM Agents in Multi-Turn Gaming Environments](https://mlsanity.com/paper/2507.11633/) (2025)
- [Information Density Principle for MLLM Benchmarks](https://mlsanity.com/paper/2503.10079/) (2025)
- [PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts](https://mlsanity.com/paper/2506.06211/) (2025)
- [SPaRC: A Spatial Pathfinding Reasoning Challenge](https://mlsanity.com/paper/2505.16686/) (2025)
- [PuzzlePlex: Benchmarking Foundation Models on Reasoning and Planning with Puzzles](https://mlsanity.com/paper/2510.06475/) (2025)
- [Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles](https://mlsanity.com/paper/2601.13705/) (2026)
- [From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models](https://mlsanity.com/paper/2511.12861/) (2025)
- [Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling](https://mlsanity.com/paper/2506.21572/) (2025)
- [FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation](https://mlsanity.com/paper/2510.08945/) (2025)
- [Physics-R1: An Audited Olympiad Corpus and Released Verifiers for Visual Physics Reasoning](https://mlsanity.com/paper/2605.14040/) (2026)

## Abstract (arXiv)

As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging multimodal problems that test a wide range of advanced reasoning and knowledge capabilities, making them a unique testbed for evaluating frontier language models. We introduce EnigmaEval, a dataset of problems and solutions derived from puzzle competitions and events that probes models' ability to perform implicit knowledge synthesis and multi-step deductive reasoning. Unlike existing reasoning and knowledge benchmarks, puzzle solving challenges models to discover hidden connections between seemingly unrelated pieces of information to uncover solution paths. The benchmark comprises 1184 puzzles of varying complexity -- each typically requiring teams of skilled solvers hours to days to complete -- with unambiguous, verifiable solutions that enable efficient evaluation. State-of-the-art language models achieve extremely low accuracy on these puzzles, even lower than other difficult benchmarks such as Humanity's Last Exam, unveiling models' shortcomings when challenged with problems requiring unstructured and lateral reasoning.