mlsanityData to 7 Oct 2026

Grounding Open-Domain Instructions to Automate Web Support Tasks

arXiv 2103.16057 ↗First posted 30 Mar 2021cs.CL cs.LGcited by 43

Not on the map. Why?

Cited by 25 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Grounding natural language instructions on the web to perform previously unseen tasks enables accessibility and automation. We introduce a task and dataset to train AI agents from open-domain, step-by-step instructions originally written for people. We build RUSS (Rapid Universal Support Service) to tackle this problem. RUSS consists of two models: First, a BERT-LSTM with pointers parses instructions to ThingTalk, a domain-specific language we design for grounding natural language on the web. Then, a grounding model retrieves the unique IDs of any webpage elements requested in ThingTalk. RUSS may interact with the user through a dialogue (e.g. ask for an address) or execute a web operation (e.g. click a button) inside the web runtime. To augment training, we synthesize natural language instructions mapped to ThingTalk. Our dataset consists of 80 different customer service problems from help websites, with a total of 741 step-by-step instructions and their corresponding actions. RUSS achieves 76.7% end-to-end accuracy predicting agent actions from single instructions. It outperforms state-of-the-art models that directly map instructions to actions without ThingTalk. Our user study shows that RUSS is preferred by actual users over web navigation.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

19 of 32 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  4. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  5. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  6. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  7. 2015Sequence to Sequence -- Video to Textcited by 157
  8. 2018Learning to Navigate in Cities Without a Mapcited by 103
  9. 2018Grounding Visual Explanationscited by 92
  10. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  11. 2019Improving Grounded Natural Language Understanding through Human-Robot Dialogcited by 25
  12. 2018Mapping Natural Language Commands to Web Elementscited by 23
7 more
  1. 2020Learning Adaptive Language Interfaces through Decompositioncited by 22
  2. 2019Genie: A Generator of Natural Language Semantic Parsers for Virtual Assistant Commandscited by 19
  3. 2020ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environmentscited by 14
  4. 2020Schema2QA: High-Quality and Low-Cost Q&A Agents for the Structured Webcited by 12
  5. 2020Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applicationscited by 3
  6. 2020Multi-Modal End-User Programming of Web-Based Virtual Assistant Skillscited by 2
  7. 2020From Natural Language Instructions to Complex Processes: Issues in Chaining Trigger Action Rulescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 6; 2024: 17; 2025: 11; 2026: 4.

43 papers.

Sort:

By area:

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  3. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  4. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  5. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  6. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  7. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  8. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  9. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  10. 2024GUI Agents: A Surveycited by 101
  11. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  12. 2024Beyond Browsing: API-Based Web Agentscited by 77
31 more
  1. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  2. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  3. 2024Large Language Models Empowered Personalized Web Agentscited by 64
  4. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  5. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  6. 2022UGIF: UI Grounded Instruction Followingcited by 25
  7. 2024WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agentscited by 22
  8. 2023WebVLN: Vision-and-Language Navigation on Websitescited by 19
  9. 2024Infogent: An Agent-Based Framework for Web Information Aggregationcited by 19
  10. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agentscited by 18
  11. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  12. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  13. 2025X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic Systemcited by 10
  14. 2024Identifying User Goals from UI Trajectoriescited by 9
  15. 2024NaviQAte: Functionality-Guided Web Application Navigationcited by 9
  16. 2025Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmarkcited by 9
  17. 2023Multi-Level Compositional Reasoning for Interactive Instruction Followingcited by 7
  18. 2025VisEscape: A Benchmark for Evaluating Exploration-driven Decision-making in Virtual Escape Roomscited by 6
  19. 2025Mind the Web: The Security of Web Use Agentscited by 5
  20. 2025V2P: Visual Attention Calibration for GUI Grounding via Background Suppression and Center Peakingcited by 4
  21. 2023RoboGPT: an intelligent agent of making embodied long-term decisions for daily instruction taskscited by 3
  22. 2023Referring to Screen Texts with Voice Assistantscited by 2
  23. 2024A Survey on Complex Tasks for Goal-Directed Interactive Agentscited by 2
  24. 2022Can Current Task-oriented Dialogue Models Automate Real-world Scenarios in the Wild?cited by 1
  25. 2025LegalWebAgent: Empowering Access to Justice via LLM-Based Web Agentscited by 1
  26. 2026Software Engineering for and with GUI Agentcited by 1
  27. 2021Procedures as Programs: Hierarchical Control of Situated Agents through Natural Languagecited by 0
  28. 2025Detecting Pipeline Failures through Fine-Grained Analysis of Web Agentscited by 0
  29. 2026V2P: Visual Attention Calibration for GUI Grounding via Background Suppression and Center Peakingcited by 0
  30. 2026See and Remember: A Multimodal Agent for Web Traversalcited by 0
  31. 2026StepGuard: Guarding Web Navigation via Single-Step Calibrationcited by 0