# SummAct: Uncovering User Intentions Through Interactive Behaviour Summarisation

arXiv: https://arxiv.org/abs/2410.08356 · HTML: https://mlsanity.com/paper/2410.08356/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2024.

## Built on by (most cited first, 9 of 9)

- [ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agent](https://mlsanity.com/paper/2507.18165/) (2025)
- [MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users](https://mlsanity.com/paper/2502.02982/) (2025)
- [Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI](https://mlsanity.com/paper/2603.01104/) (2026)
- [Bi-Fact: A Bidirectional Factorization-based Evaluation of Intent Extraction from UI Trajectories](https://mlsanity.com/paper/2502.13149/) (2025)
- [Small Models, Big Results: Achieving Superior Intent Extraction through Decomposition](https://mlsanity.com/paper/2509.12423/) (2025)
- [Fair Representation in Parliamentary Summaries: Measuring and Mitigating Inclusion Bias](https://mlsanity.com/paper/2507.14221/) (2025)
- [When Should an AI Act? A Human-Centered Model of Scene, Context, and Behavior for Agentic AI Design](https://mlsanity.com/paper/2602.22814/) (2026)
- ["What Are You Really Trying to Do?": Co-Creating Life Goals from Everyday Computer Use](https://mlsanity.com/paper/2605.00497/) (2026)
- [Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay](https://mlsanity.com/paper/2608.05784/) (2026)

## Abstract (arXiv)

Recent work has highlighted the potential of modelling interactive behaviour analogously to natural language. We propose interactive behaviour summarisation as a novel computational task and demonstrate its usefulness for automatically uncovering latent user intentions while interacting with graphical user interfaces. To tackle this task, we introduce SummAct, a novel hierarchical method to summarise low-level input actions into high-level intentions. SummAct first identifies sub-goals from user actions using a large language model and in-context learning. High-level intentions are then obtained by fine-tuning the model using a novel UI element attention to preserve detailed context information embedded within UI elements during summarisation. Through a series of evaluations, we demonstrate that SummAct significantly outperforms baselines across desktop and mobile interfaces as well as interactive tasks by up to 21.9%. We further show three exciting interactive applications benefited from SummAct: interactive behaviour forecasting, automatic behaviour synonym identification, and language-based behaviour retrieval.