# Small Models, Big Results: Achieving Superior Intent Extraction through Decomposition

arXiv: https://arxiv.org/abs/2509.12423 · HTML: https://mlsanity.com/paper/2509.12423/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 15 Sep 2025.

## Builds on (most cited first, 18 of 18)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [Aria-UI: Visual Grounding for GUI Instructions](https://mlsanity.com/paper/2412.16256/) (2024)
- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [GUI Agents with Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2411.04890/) (2024)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [UIBert: Learning Generic Multimodal Representations for UI Understanding](https://mlsanity.com/paper/2107.13731/) (2021)
- [Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus](https://mlsanity.com/paper/2209.14927/) (2022)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [Hierarchical Multimodal Transformer to Summarize Videos](https://mlsanity.com/paper/2109.10559/) (2021)
- [Enhancing Long Video Understanding via Hierarchical Event-Based Memory](https://mlsanity.com/paper/2409.06299/) (2024)
- [Identifying User Goals from UI Trajectories](https://mlsanity.com/paper/2406.14314/) (2024)
- [SummAct: Uncovering User Intentions Through Interactive Behaviour Summarisation](https://mlsanity.com/paper/2410.08356/) (2024)
- [Bi-Fact: A Bidirectional Factorization-based Evaluation of Intent Extraction from UI Trajectories](https://mlsanity.com/paper/2502.13149/) (2025)

## Built on by (most cited first, 1 of 1)

- [PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents](https://mlsanity.com/paper/2603.08013/) (2026)

## Abstract (arXiv)

Understanding user intents from UI interaction trajectories remains a challenging, yet crucial, frontier in intelligent agent development. While massive, datacenter-based, multi-modal large language models (MLLMs) possess greater capacity to handle the complexities of such sequences, smaller models which can run on-device to provide a privacy-preserving, low-cost, and low-latency user experience, struggle with accurate intent inference. We address these limitations by introducing a novel decomposed approach: first, we perform structured interaction summarization, capturing key information from each user action. Second, we perform intent extraction using a fine-tuned model operating on the aggregated summaries. This method improves intent understanding in resource-constrained models, even surpassing the base performance of large MLLMs.