# GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection

arXiv: https://arxiv.org/abs/2512.09396 · HTML: https://mlsanity.com/paper/2512.09396/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 10 Dec 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024)
- [GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices](https://mlsanity.com/paper/2406.08451/) (2024)
- [InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners](https://mlsanity.com/paper/2504.14239/) (2025)
- [GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation](https://mlsanity.com/paper/2311.07562/) (2023)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)
- [InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection](https://mlsanity.com/paper/2501.04575/) (2025)
- [Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents](https://mlsanity.com/paper/2412.13194/) (2024)
- [CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers](https://mlsanity.com/paper/2412.13810/) (2024)
- [MobileFlow: A Multimodal LLM For Mobile GUI Agent](https://mlsanity.com/paper/2407.04346/) (2024)
- [VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling](https://mlsanity.com/paper/2112.05692/) (2021)
- [UGIF: UI Grounded Instruction Following](https://mlsanity.com/paper/2211.07615/) (2022)
- [UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis](https://mlsanity.com/paper/2504.11257/) (2025)
- [Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API](https://mlsanity.com/paper/2310.04716/) (2023)
- [MobileAgent: enhancing mobile control via human-machine interaction and SOP integration](https://mlsanity.com/paper/2401.04124/) (2024)
- [WebQuest: A Benchmark for Multimodal QA on Web Page Sequences](https://mlsanity.com/paper/2409.13711/) (2024)
- [EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data](https://mlsanity.com/paper/2410.19461/) (2024)
- [Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining](https://mlsanity.com/paper/2412.10342/) (2024)
- [MobileViews: A Million-scale and Diverse Mobile GUI Dataset](https://mlsanity.com/paper/2409.14337/) (2024)

## Abstract (arXiv)

Building AI systems for GUI automation task has attracted remarkable research efforts, where MLLMs are leveraged for processing user requirements and give operations. However, GUI automation includes a wide range of tasks, from document processing to online shopping, from CAD to video editing. Diversity between particular tasks requires MLLMs for GUI automation to have heterogeneous capabilities and master multidimensional expertise, raising problems on constructing such a model. To address such challenge, we propose GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection, a novel MLLM-based GUI automation agent framework designed for integrating knowledge and combining capabilities from heterogeneous models to build GUI automation agent systems with higher performance. Since different GUI-specific MLLMs are trained on different dataset and thus have different strengths, GAIR introduced a general-purpose MLLM for jointly processing the information from multiple GUI-specific models, further enhancing performance of the agent framework. The general-purpose MLLM also serves as decision maker, trying to execute a reasonable operation based on previously gathered information. When the general-purpose model thinks that there isn't sufficient information for a reasonable decision, GAIR would transit into group reflection status, where the general-purpose model would provide GUI-specific models with different instructions and hints based on their strengths and weaknesses, driving them to gather information with more significance and accuracy that can support deeper reasoning and decision. We evaluated the effectiveness and reliability of GAIR through extensive experiments on GUI benchmarks.