# E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion

arXiv: https://arxiv.org/abs/2406.14250 · HTML: https://mlsanity.com/paper/2406.14250/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018)
- [You Only Look at Screens: Multimodal Chain-of-Action Agents](https://mlsanity.com/paper/2309.11436/) (2023)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation](https://mlsanity.com/paper/2311.07562/) (2023)
- [UIBert: Learning Generic Multimodal Representations for UI Understanding](https://mlsanity.com/paper/2107.13731/) (2021)
- [A data-driven approach for learning to control computers](https://mlsanity.com/paper/2202.08137/) (2022)
- [XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understanding](https://mlsanity.com/paper/2203.06947/) (2022)
- [Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation](https://mlsanity.com/paper/2312.03003/) (2023)
- [DroidBot-GPT: GPT-powered UI Automation for Android](https://mlsanity.com/paper/2304.07061/) (2023)
- [A Zero-Shot Language Agent for Computer Control with Structured Reflection](https://mlsanity.com/paper/2310.08740/) (2023)
- [Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API](https://mlsanity.com/paper/2310.04716/) (2023)
- [Mobile User Interface Element Detection Via Adaptively Prompt Tuning](https://mlsanity.com/paper/2305.09699/) (2023)
- [Learning UI Navigation through Demonstrations composed of Macro Actions](https://mlsanity.com/paper/2110.08653/) (2021)

## Built on by (most cited first, 11 of 11)

- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [UI-Venus Technical Report: Building High-performance UI Agents with RFT](https://mlsanity.com/paper/2508.10833/) (2025)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents](https://mlsanity.com/paper/2504.12679/) (2025)
- [UI-Venus-1.5 Technical Report](https://mlsanity.com/paper/2602.09082/) (2026)
- [UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience](https://mlsanity.com/paper/2603.24533/) (2026)
- [TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments](https://mlsanity.com/paper/2505.17629/) (2025)
- [VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Tasks](https://mlsanity.com/paper/2512.16501/) (2025)
- [Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization](https://mlsanity.com/paper/2602.13653/) (2026)
- [UIPress: Bringing Optical Token Compression to UI-to-Code Generation](https://mlsanity.com/paper/2604.09442/) (2026)
- [Software Engineering for and with GUI Agent](https://mlsanity.com/paper/2608.09278/) (2026)

## Abstract (arXiv)

Online GUI navigation on mobile devices has driven a lot of attention recent years since it contributes to many real-world applications. With the rapid development of large language models (LLM), multimodal large language models (MLLM) have tremendous potential on this task. However, existing MLLMs need high quality data to improve its abilities of making the correct navigation decisions according to the human user inputs. In this paper, we developed a novel and highly valuable dataset, named E-ANT, as the first Chinese GUI navigation dataset that contains real human behaviour and high quality screenshots with annotations, containing nearly 40,000 real human traces over 5000+ different tinyAPPs. Furthermore, we evaluate various powerful MLLMs on E-ANT and show their experiments results with sufficient ablations. We believe that our proposed dataset will be beneficial for both the evaluation and development of GUI navigation and LLM/MLLM decision-making capabilities.