# Mapping Natural Language Commands to Web Elements

arXiv: https://arxiv.org/abs/1808.09132 · HTML: https://mlsanity.com/paper/1808.09132/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2018.

## Built on by (most cited first, 23 of 23)

- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI](https://mlsanity.com/paper/2205.11029/) (2022)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [DroidBot-GPT: GPT-powered UI Automation for Android](https://mlsanity.com/paper/2304.07061/) (2023)
- [ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation](https://mlsanity.com/paper/2312.13108/) (2023)
- [Grounding Open-Domain Instructions to Automate Web Support Tasks](https://mlsanity.com/paper/2103.16057/) (2021)
- [OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use](https://mlsanity.com/paper/2508.04482/) (2025)
- [Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments](https://mlsanity.com/paper/2104.08560/) (2021)
- [VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling](https://mlsanity.com/paper/2112.05692/) (2021)
- [Tur[k]ingBench: A Challenge Benchmark for Web Agents](https://mlsanity.com/paper/2403.11905/) (2024)
- [Grounding Natural Language Instructions: Can Large Language Models Capture Spatial Information?](https://mlsanity.com/paper/2109.08634/) (2021)
- [Lexi: Self-Supervised Learning of the UI Language](https://mlsanity.com/paper/2301.10165/) (2023)
- [MUG: Interactive Multimodal Grounding on User Interfaces](https://mlsanity.com/paper/2209.15099/) (2022)
- [Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces](https://mlsanity.com/paper/2404.13521/) (2024)
- [DiLogics: Creating Web Automation Programs With Diverse Logics](https://mlsanity.com/paper/2308.05828/) (2023)
- [Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning](https://mlsanity.com/paper/2405.00516/) (2024)
- [GUI Agents with Reinforcement Learning: Toward Digital Inhabitants](https://mlsanity.com/paper/2604.27955/) (2026)
- [Referring to Screen Texts with Voice Assistants](https://mlsanity.com/paper/2306.07298/) (2023)
- [Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthrough](https://mlsanity.com/paper/2512.03568/) (2025)
- [GhostUI: Unveiling Hidden Interactions in Mobile UI](https://mlsanity.com/paper/2601.19258/) (2026)

## Abstract (arXiv)

The web provides a rich, open-domain environment with textual, structural, and spatial properties. We propose a new task for grounding language in this environment: given a natural language command (e.g., "click on the second article"), choose the correct element on the web page (e.g., a hyperlink or text box). We collected a dataset of over 50,000 commands that capture various phenomena such as functional references (e.g. "find who made this site"), relational reasoning (e.g. "article by john"), and visual reasoning (e.g. "top-most article"). We also implemented and analyzed three baseline models that capture different phenomena present in the dataset.