# Visual Large Language Models Exhibit Human-Level Cognitive Flexibility in the Wisconsin Card Sorting Test

arXiv: https://arxiv.org/abs/2505.22112 · HTML: https://mlsanity.com/paper/2505.22112/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 51 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 16 of 16)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Vision language models are blind: Failing to translate detailed visual features into words](https://mlsanity.com/paper/2407.06581/) (2024)
- [M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought](https://mlsanity.com/paper/2405.16473/) (2024)
- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [Evaluating Cognitive Maps and Planning in Large Language Models with CogEval](https://mlsanity.com/paper/2309.15129/) (2023)
- [Large Language Models are Visual Reasoning Coordinators](https://mlsanity.com/paper/2310.15166/) (2023)
- [Integration of cognitive tasks into artificial general intelligence test for large models](https://mlsanity.com/paper/2402.02547/) (2024)

## Built on by (most cited first, 5 of 5)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilities](https://mlsanity.com/paper/2603.02540/) (2026)
- [Evaluating Cognitive Age Alignment in Interactive AI Agents](https://mlsanity.com/paper/2605.17894/) (2026)
- [Set-shifting Behavioral Test for Harnessed Agents](https://mlsanity.com/paper/2607.13396/) (2026)
- [A Comprehensive Review of Generative Physical Artificial Intelligence](https://mlsanity.com/paper/2609.18111/) (2026)

## Abstract (arXiv)

Cognitive flexibility has been extensively studied in human cognition but remains relatively unexplored in the context of Visual Large Language Models (VLLMs). This study assesses the cognitive flexibility of state-of-the-art VLLMs (GPT-4o, Gemini-1.5 Pro, and Claude-3.5 Sonnet) using the Wisconsin Card Sorting Test (WCST), a classic measure of set-shifting ability. Our results reveal that VLLMs achieve or surpass human-level set-shifting capabilities under chain-of-thought prompting with text-based inputs. However, their abilities are highly influenced by both input modality and prompting strategy. In addition, we find that through role-playing, VLLMs can simulate various functional deficits aligned with patients having impairments in cognitive flexibility, suggesting that VLLMs may possess a cognitive architecture, at least regarding the ability of set-shifting, similar to the brain. This study reveals the fact that VLLMs have already approached the human level on a key component underlying our higher cognition, and highlights the potential to use them to emulate complex brain processes.