mlsanityData to 7 Oct 2026

InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection

arXiv 2501.04575 ↗First posted 8 Jan 2025cs.AI cs.CL cs.HCcited by 67

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Graphical User Interface (GUI) Agents, powered by multimodal large language models (MLLMs), have shown great potential for task automation on computing devices such as computers and mobile phones. However, existing agents face challenges in multi-step reasoning and reliance on textual annotations, limiting their effectiveness. We introduce textitInfiGUIAgent, an MLLM-based GUI Agent trained with a two-stage supervised fine-tuning pipeline. Stage 1 enhances fundamental skills such as GUI understanding and grounding, while Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning skills using synthesized data to enable native reasoning abilities of the agents. textitInfiGUIAgent achieves competitive performance on several GUI benchmarks, highlighting the impact of native reasoning skills in enhancing GUI interaction for automation tasks. Resources are available at urlhttps://github.com/Reallm-Labs/InfiGUIAgent.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 67 references that are in the map. First posted on arXiv 8 Jan 2025.

Builds on

49 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
36 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  3. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  4. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  5. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  6. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  7. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  8. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  9. 2024Understanding the planning of LLM agents: A surveycited by 414
  10. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  11. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  12. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  13. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  14. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  15. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  16. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  17. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  18. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  19. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  20. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  21. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 115
  22. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  23. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  24. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  25. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  26. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  27. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agentscited by 72
  28. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 65
  29. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 61
  30. 2024MobileFlow: A Multimodal LLM For Mobile GUI Agentcited by 31
  31. 2022One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Codecited by 26
  32. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24
  33. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversationscited by 17
  34. 2024Leveraging Print Debugging to Improve Code Generation in Large Language Modelscited by 15
  35. 2024GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learningcited by 13
  36. 2024InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 40; 2026: 27.

67 papers.

Sort:

By area:

  1. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  2. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  4. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  5. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  6. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  7. 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanismcited by 25
  8. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 25
  9. 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
  10. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
  11. 2025Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attackscited by 21
  12. 2026SkillTrojan: Backdoor Attacks on Skill-Based Agent Systemscited by 21
38 more
  1. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 20
  2. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  3. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
  4. 2025AppVLM: A Lightweight Vision Language Model for Online App Controlcited by 15
  5. 2026Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correctioncited by 14
  6. 2025Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalizationcited by 11
  7. 2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling Systemcited by 11
  8. 2025History-Aware Reasoning for GUI Agentscited by 11
  9. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 11
  10. 2026Adaptive Milestone Reward for GUI Agentscited by 11
  11. 2025HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimizationcited by 10
  12. 2025MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operationcited by 9
  13. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agentscited by 9
  14. 2025Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Questionscited by 8
  15. 2025GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agentscited by 8
  16. 2025Agentic Lybic: Multi-Agent Execution System with Tiered Reasoning and Orchestrationcited by 8
  17. 2025GUI-PRA: Process Reward Agent for GUI Taskscited by 8
  18. 2025UITron-Speech: Towards Automated GUI Agents Based on Speech Instructionscited by 7
  19. 2025Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphscited by 7
  20. 2025EcoAgent: An Efficient Device-Cloud Collaborative Multi-Agent Framework for Mobile Automationcited by 6
  21. 2026Towards Pixel-Level VLM Perception via Simple Points Predictioncited by 5
  22. 2026On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Lengthcited by 5
  23. 2025Towards Pervasive Distributed Agentic Generative AI -- A State of The Artcited by 4
  24. 2025See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Togglescited by 4
  25. 2026MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environmentcited by 4
  26. 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbationscited by 3
  27. 2025Invisible to Humans, Triggered by Agents: Stealthy Jailbreak Attacks on Mobile Vision-Language Agentscited by 3
  28. 2025GUI-Rise: Structured Reasoning and History Summarization for GUI Navigationcited by 3
  29. 2025Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agentscited by 3
  30. 2026POINTS-GUI-G: GUI-Grounding Journeycited by 3
  31. 2026CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuningcited by 3
  32. 2026CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agentscited by 3
  33. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  34. 2026Self-evolving LLM agents with in-distribution Optimizationcited by 3
  35. 2025DPO Learning with LLMs-Judge Signal for Computer Use Agentscited by 2
  36. 2026GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Modelscited by 2
  37. 2026ST-Lite: Training-Free KV Cache Compression with Spatio-Trajectory Guidance for Long-Horizon GUI Agentscited by 2
  38. 2026K^2-Agent: Co-Evolving Know-What and Know-How for Hierarchical Mobile Device Controlcited by 2

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
  2. 2026UI-Venus-2 Technical Report
  3. 2024GUI Agents with Foundation Models: A Comprehensive Survey
  4. 2024Large Language Model-Brained GUI Agents: A Survey
  5. 2026Towards Scalable Lightweight GUI Agents via Multi-role Orchestration
  6. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
  7. 2025Learning, Reasoning, Refinement: A Framework for Kahneman's Dual-System Intelligence in GUI Agents
  8. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Users