# Recurrent Latent Visual Search for GUI Grounding

arXiv: https://arxiv.org/abs/2610.05185 · HTML: https://mlsanity.com/paper/2610.05185/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 39 of its 46 references that are in the map.
- First posted on arXiv 4 Oct 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 30 of 39)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning](https://mlsanity.com/paper/2403.16999/) (2024)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [ShowUI: One Vision-Language-Action Model for GUI Visual Agent](https://mlsanity.com/paper/2411.17465/) (2024)
- [Compressed Chain of Thought: Efficient Reasoning Through Dense Representations](https://mlsanity.com/paper/2412.13171/) (2024)
- [On the Effects of Data Scale on UI Control Agents](https://mlsanity.com/paper/2406.03679/) (2024)
- [Aria-UI: Visual Grounding for GUI Instructions](https://mlsanity.com/paper/2412.16256/) (2024)
- [Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis](https://mlsanity.com/paper/2505.13227/) (2025)
- [Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens](https://mlsanity.com/paper/2506.17218/) (2025)
- [Latent Visual Reasoning](https://mlsanity.com/paper/2509.24251/) (2025)
- [Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models](https://mlsanity.com/paper/2503.20752/) (2025)
- [GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents](https://mlsanity.com/paper/2506.03143/) (2025)
- [Monet: Reasoning in Latent Visual Space Beyond Images and Language](https://mlsanity.com/paper/2511.21395/) (2025)
- [UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction](https://mlsanity.com/paper/2503.15661/) (2025)
- [Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens](https://mlsanity.com/paper/2511.19418/) (2025)
- [GUI-G2: Gaussian Reward Modeling for GUI Grounding](https://mlsanity.com/paper/2507.15846/) (2025)
- [LLM Latent Reasoning as Chain of Superposition](https://mlsanity.com/paper/2510.15522/) (2025)
- [DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning](https://mlsanity.com/paper/2507.00008/) (2025)
- [Visual Test-time Scaling for GUI Agent Grounding](https://mlsanity.com/paper/2505.00684/) (2025)
- [InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimization](https://mlsanity.com/paper/2508.05731/) (2025)
- [R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding](https://mlsanity.com/paper/2507.05673/) (2025)
- [UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning](https://mlsanity.com/paper/2510.20286/) (2025)
- [Grounding Computer Use Agents on Human Demonstrations](https://mlsanity.com/paper/2511.07332/) (2025)
- [MVP: Multiple View Prediction Improves GUI Grounding](https://mlsanity.com/paper/2512.08529/) (2025)
- [Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding](https://mlsanity.com/paper/2605.00642/) (2026)
- [textscGUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding](https://mlsanity.com/paper/2510.04039/) (2025)
- [GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents](https://mlsanity.com/paper/2601.09770/) (2026)

## Abstract (arXiv)

GUI grounding is a critical capability for GUI agents powered by vision-language models, helping them execute user instructions by locating the corresponding elements in screenshots. Single-step grounding struggles with small elements and dense layouts, motivating multi-step visual search. However, existing approaches commonly rely on textual reasoning misaligned with visual space or costly multi-round interactions with external visual tools. To make multi-step visual search an explicit spatial process within the model, we propose ReLaViS, which performs Recurrent Latent Visual Search in a single interaction round. At each step, a spatial search head uses the hidden state to query the screenshot's visual tokens, producing a spatial search distribution that explicitly represents the search focus. This distribution then aggregates the visual tokens into latent visual evidence, which is recurrently fed back as the next input embedding to condition subsequent search. We further introduce a GUI-aware coarse-to-fine inductive bias through trajectories constructed from flat element annotations, supervising search from the global interface through intermediate element groups to the target. Built on Qwen2.5-VL-7B, ReLaViS improves ScreenSpot-Pro accuracy by 3.1 percentage points to 56.3% with only a 3.5% increase in inference FLOPs and outperforms the matched single-step baseline on all five benchmarks.