mlsanityData to 7 Oct 2026

TinyClick: Single-Turn Agent for Empowering GUI Automation

arXiv 2410.11871 ↗First posted 9 Oct 2024cs.HC cs.AIcited by 12

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

We present an UI agent for user interface (UI) interaction tasks, using Vision-Language Model Florence-2-Base. The agent's primary task is identifying the screen coordinates of the UI element corresponding to the user's command. It demonstrates very strong performance on Screenspot and OmniAct annotations, while maintaining a very small size of 0.27B parameters and minimal latency. Moreover, training needs small compute budget of 56 GPU-hours (worth about 40 USD). Relevant improvement comes from vision-specific multi-task training and MLLM-based data augmentation. We hope that decreased needs for expensive compute resources and manually annotated data will allow to facilitate more inclusive and sustainable research of UI agents.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 46 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

33 of 46 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  2. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents3 Jul
  3. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
  4. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar
  5. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web27 Feb
  6. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  7. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  8. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
  9. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  10. 2023You Only Look at Screens: Multimodal Chain-of-Action Agents20 Sep

Other references · 23

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  6. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  7. 2018Albumentations: fast and flexible image augmentationscited by 558
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  9. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  10. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  11. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  12. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
11 more
  1. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  2. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  3. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  4. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  5. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  6. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 65
  7. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  8. 2018Learning to Navigate the Webcited by 60
  9. 2024Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Groundingcited by 18
  10. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12
  11. 2024V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLMcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 3.

12 papers.

  1. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  4. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 44
  5. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  6. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  7. 2026Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agentscited by 17
  8. 2026Agentic Test-Time Scaling for WebAgentscited by 7
  9. 2025LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmarkcited by 3
  10. 2025Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UIcited by 1
  11. 2025GUIrilla: A Scalable Framework for Automated Desktop UI Explorationcited by 0
  12. 2026GPA: Learning GUI Process Automation from Demonstrationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025UIPro: Unleashing Superior Interaction Capability For GUI Agents
  2. 2024GUI Agents: A Survey
  3. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
  4. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents
  5. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  6. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
  7. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
  8. 2025SpiritSight Agent: Advanced GUI Agent with One Look