# Learning, Reasoning, Refinement: A Framework for Kahneman's Dual-System Intelligence in GUI Agents arXiv: https://arxiv.org/abs/2506.17913 ยท HTML: https://mlsanity.com/paper/2506.17913/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. - Builds on 27 of its 41 references that are in the map. - First posted on arXiv 22 Jun 2025. ## Placed in - LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/) - LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/) ## Builds on (most cited first, 27 of 27) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023) - [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023) - [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024) - [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025) - [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024) - [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024) - [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024) - [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025) - [GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents](https://mlsanity.com/paper/2504.10458/) (2025) - [ShowUI: One Vision-Language-Action Model for GUI Visual Agent](https://mlsanity.com/paper/2411.17465/) (2024) - [Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration](https://mlsanity.com/paper/2406.01014/) (2024) - [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020) - [UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning](https://mlsanity.com/paper/2503.21620/) (2025) - [Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models](https://mlsanity.com/paper/2310.06117/) (2023) - [A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce](https://mlsanity.com/paper/2504.11343/) (2025) - [Aria-UI: Visual Grounding for GUI Instructions](https://mlsanity.com/paper/2412.16256/) (2024) - [ScreenAgent: A Vision Language Model-driven Computer Control Agent](https://mlsanity.com/paper/2402.07945/) (2024) - [Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements](https://mlsanity.com/paper/2010.04295/) (2020) - [META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI](https://mlsanity.com/paper/2205.11029/) (2022) - [Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory](https://mlsanity.com/paper/2311.08719/) (2023) - [PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC](https://mlsanity.com/paper/2502.14282/) (2025) - [OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models](https://mlsanity.com/paper/2502.16161/) (2025) ## Built on by (most cited first, 2 of 2) - [GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoning](https://mlsanity.com/paper/2509.15738/) (2025) - [SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement](https://mlsanity.com/paper/2507.01643/) (2025) ## Abstract (arXiv) Graphical User Interface (GUI) agents have made significant progress in automating digital tasks through the utilization of computer vision and language models. Nevertheless, existing agent systems encounter notable limitations. Firstly, they predominantly depend on trial and error decision making rather than progressive reasoning, thereby lacking the capability to learn and adapt from interactive encounters. Secondly, these systems are assessed using overly simplistic single step accuracy metrics, which do not adequately reflect the intricate nature of real world GUI interactions. In this paper, we present CogniGUI, a cognitive framework developed to overcome these limitations by enabling adaptive learning for GUI automation resembling human-like behavior. Inspired by Kahneman's Dual Process Theory, our approach combines two main components: (1) an omni parser engine that conducts immediate hierarchical parsing of GUI elements through quick visual semantic analysis to identify actionable components, and (2) a Group based Relative Policy Optimization (GRPO) grounding agent that assesses multiple interaction paths using a unique relative reward system, promoting minimal and efficient operational routes. This dual-system design facilitates iterative ''exploration learning mastery'' cycles, enabling the agent to enhance its strategies over time based on accumulated experience. Moreover, to assess the generalization and adaptability of agent systems, we introduce ScreenSeek, a comprehensive benchmark that includes multi application navigation, dynamic state transitions, and cross interface coherence, which are often overlooked challenges in current benchmarks. Experimental results demonstrate that CogniGUI surpasses state-of-the-art methods in both the current GUI grounding benchmarks and our newly proposed benchmark.