# Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation

arXiv: https://arxiv.org/abs/2506.02648 · HTML: https://mlsanity.com/paper/2506.02648/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 23 of 23)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems](https://mlsanity.com/paper/2505.11831/) (2025)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [O1 Replication Journey: A Strategic Progress Report -- Part 1](https://mlsanity.com/paper/2410.18982/) (2024)
- [CodeGen2: Lessons for Training LLMs on Programming and Natural Languages](https://mlsanity.com/paper/2305.02309/) (2023)
- [Hypothesis Search: Inductive Reasoning with Language Models](https://mlsanity.com/paper/2309.05660/) (2023)
- [ARC Prize 2024: Technical Report](https://mlsanity.com/paper/2412.04604/) (2024)
- [DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks](https://mlsanity.com/paper/2309.17167/) (2023)
- [O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?](https://mlsanity.com/paper/2411.16489/) (2024)
- [LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations](https://mlsanity.com/paper/2305.18354/) (2023)
- [NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes](https://mlsanity.com/paper/2312.14890/) (2023)
- [SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines](https://mlsanity.com/paper/2502.14739/) (2025)
- [Dynamic Evaluation of Large Language Models by Meta Probing Agents](https://mlsanity.com/paper/2402.14865/) (2024)
- [An Open Source Data Contamination Report for Large Language Models](https://mlsanity.com/paper/2310.17589/) (2023)
- [H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark](https://mlsanity.com/paper/2409.01374/) (2024)
- [Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping](https://mlsanity.com/paper/2410.08695/) (2024)
- [The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding](https://mlsanity.com/paper/2502.08946/) (2025)
- [Speak It Out: Solving Symbol-Related Problems with Symbol-to-Language Conversion for Language Models](https://mlsanity.com/paper/2401.11725/) (2024)
- [Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Task](https://mlsanity.com/paper/2502.07190/) (2025)

## Built on by (most cited first, 6 of 6)

- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilities](https://mlsanity.com/paper/2603.02540/) (2026)
- [Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problems](https://mlsanity.com/paper/2602.06319/) (2026)
- [Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoning](https://mlsanity.com/paper/2508.10057/) (2025)
- [Disentangling generalization and memorization in large language models using chess](https://mlsanity.com/paper/2601.16823/) (2026)
- [Looking Inside LLMs: Small-World Connectivity as a Signature of Reasoning Performance](https://mlsanity.com/paper/2610.12304/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have demonstrated impressive reasoning capacities that mirror human-like thinking. However, whether LLMs possess genuine fluid intelligence (i.e., the ability to reason abstractly and generalize rules in novel situations) remains an open question. Existing reasoning benchmarks either focus on domain-specific knowledge (crystallized intelligence) or lack interpretability. To address these limitations, we propose DRE-Bench, a dynamic reasoning evaluation benchmark grounded in a hierarchical cognitive framework. DRE-Bench consists of 36 abstract reasoning tasks organized across four cognitive levels, with each task featuring multiple dynamic variants that test the same underlying latent rule. This design enables fine-grained, interpretable, and reliable assessments of fluid intelligence. We evaluate a range of state-of-the-art LLMs, including both general LLMs (GPT-4o, Claude 3.7) and reasoning LLMs (o1, DeepSeek-R1, QwQ, Skywork-OR1). Experimental results reveal that although most LLMs achieve competent and robust performance in low-level cognition, they struggle with high-level cognition and exhibit limited generalization as task complexity grows. Our findings highlight the gap between current LLMs and true human-like fluid intelligence and offer a new path for systematically tracking reasoning progress in LLMs.