mlsanityData to 7 Oct 2026

CogAgent: A Visual Language Model for GUI Agents

arXiv 2312.08914 ↗First posted 14 Dec 2023cs.CVcited by 692

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

People are spending an enormous amount of time on digital devices through graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large language models (LLMs) such as ChatGPT can assist people in tasks like writing emails, but struggle to understand and interact with GUIs, thus limiting their potential to increase automation levels. In this paper, we introduce CogAgent, an 18-billion-parameter visual language model (VLM) specializing in GUI understanding and navigation. By utilizing both low-resolution and high-resolution image encoders, CogAgent supports input at a resolution of 1120*1120, enabling it to recognize tiny page elements and text. As a generalist visual language model, CogAgent achieves the state of the art on five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA, Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using only screenshots as input, outperforms LLM-based methods that consume extracted HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW, advancing the state of the art. The model and codes are available at https://github.com/THUDM/CogVLM, with a new version of CogAgent-9B-20241220 available at https://github.com/THUDM/CogAgent.

Key facts. Cited by 692 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 42 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

33 of 42 references.

References · 33

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
21 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  6. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  7. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2021OCR-free Document Understanding Transformercited by 400
  10. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  11. 2019Scene Text Visual Question Answeringcited by 327
  12. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  13. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  14. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  15. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  16. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  17. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  18. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  19. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  20. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 147
  21. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 198; 2025: 280; 2026: 211.

692 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  6. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  7. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  8. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  9. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  10. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  11. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  12. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
38 more
  1. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  2. 2024What matters when building vision-language models?cited by 321
  3. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  4. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  5. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  6. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  7. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  8. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  9. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  10. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  11. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  12. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  13. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  14. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  15. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  16. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  17. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  18. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  19. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  20. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  21. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  22. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  23. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  24. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  25. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  26. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  27. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  28. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  29. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  30. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  31. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  32. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
  33. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  34. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  35. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  36. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  37. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  38. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149

50 most cited of 692 All 692 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
  2. 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
  3. 2026GUITrans2Act: Understanding User Operational Behaviors from Mobile GUI Interactions with Vision-Language Models
  4. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understanding
  5. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
  6. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  7. 2024GUI Agents with Foundation Models: A Comprehensive Survey
  8. 2025Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation