mlsanityData to 7 Oct 2026

Mapping Natural Language Commands to Web Elements

arXiv 1808.09132 ↗First posted 28 Aug 2018cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

The web provides a rich, open-domain environment with textual, structural, and spatial properties. We propose a new task for grounding language in this environment: given a natural language command (e.g., "click on the second article"), choose the correct element on the web page (e.g., a hyperlink or text box). We collected a dataset of over 50,000 commands that capture various phenomena such as functional references (e.g. "find who made this site"), relational reasoning (e.g. "article by john"), and visual reasoning (e.g. "top-most article"). We also implemented and analyzed three baseline models that capture different phenomena present in the dataset.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 5; 2023: 6; 2024: 4; 2025: 2; 2026: 2.

23 papers.

Sort:

By area:

  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  2. 2023Language Models can Solve Computer Taskscited by 371
  3. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  4. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  5. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  6. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  7. 2023DroidBot-GPT: GPT-powered UI Automation for Androidcited by 47
  8. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  9. 2021Grounding Open-Domain Instructions to Automate Web Support Taskscited by 43
  10. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  11. 2021Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environmentscited by 28
  12. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
11 more
  1. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agentscited by 18
  2. 2021Grounding Natural Language Instructions: Can Large Language Models Capture Spatial Information?cited by 10
  3. 2023Lexi: Self-Supervised Learning of the UI Languagecited by 10
  4. 2022MUG: Interactive Multimodal Grounding on User Interfacescited by 9
  5. 2024Graph4GUI: Graph Neural Networks for Representing Graphical User Interfacescited by 6
  6. 2023DiLogics: Creating Web Automation Programs With Diverse Logicscited by 5
  7. 2024Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learningcited by 4
  8. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  9. 2023Referring to Screen Texts with Voice Assistantscited by 2
  10. 2025Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthroughcited by 1
  11. 2026GhostUI: Unveiling Hidden Interactions in Mobile UIcited by 1