mlsanityData to 7 Oct 2026

A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility

arXiv 2202.02312 ↗First posted 4 Feb 2022cs.CL cs.CV cs.HCcited by 61

Not on the map. Why?

Cited by 22 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision-language navigation (VLN), in which an agent follows language instruction in a visual environment, has been studied under the premise that the input command is fully feasible in the environment. Yet in practice, a request may not be possible due to language ambiguity or environment changes. To study VLN with unknown command feasibility, we introduce a new dataset Mobile app Tasks with Iterative Feedback (MoTIF), where the goal is to complete a natural language command in a mobile app. Mobile apps provide a scalable domain to study real downstream uses of VLN methods. Moreover, mobile app commands provide instruction for interactive navigation, as they result in action sequences with state changes via clicking, typing, or swiping. MoTIF is the first to include feasibility annotations, containing both binary feasibility labels and fine-grained labels for why tasks are unsatisfiable. We further collect follow-up questions for ambiguous queries to enable research on task uncertainty resolution. Equipped with our dataset, we propose the new problem of feasibility prediction, in which a natural language instruction and multimodal app environment are used to predict command feasibility. MoTIF provides a more realistic app dataset as it contains many diverse environments, high-level goals, and longer action sequences than prior work. We evaluate interactive VLN methods using MoTIF, quantify the generalization ability of current approaches to new app environments, and measure the effect of task feasibility on navigation performance.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 4 Feb 2022.

Builds on

32 of 46 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  9. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  11. 2017Word Translation Without Parallel Datacited by 480
  12. 2018VirtualHome: Simulating Household Activities via Programscited by 411
20 more
  1. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  2. 2017Embodied Question Answeringcited by 335
  3. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  4. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  5. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  6. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  7. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  8. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  9. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  10. 2021CanvasVAE: Learning to Generate Vector Graphic Documentscited by 93
  11. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  12. 2021Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigationcited by 47
  13. 2017Visual Semantic Planning using Deep Successor Representationscited by 45
  14. 2021Screen2Vec: Semantic Embedding of GUI Screens and GUI Componentscited by 41
  15. 2020Factorizing Perception and Policy for Interactive Instruction Followingcited by 28
  16. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  17. 2018Mapping Natural Language Commands to Web Elementscited by 23
  18. 2016Question Relevance in VQA: Identifying Non-Visual And False-Premise Questionscited by 16
  19. 2018Visual Dialogue without Vision or Dialoguecited by 6
  20. 2020Determining Question-Answer Plausibility in Crowdsourced Datasets Using Multi-Task Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 11; 2024: 21; 2025: 21; 2026: 4.

61 papers.

Sort:

By area:

  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  2. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  3. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  4. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  5. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  6. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  7. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  8. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  9. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  10. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101
  11. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  12. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
38 more
  1. 2024Beyond Browsing: API-Based Web Agentscited by 77
  2. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  3. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 65
  4. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  5. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  6. 2023DroidBot-GPT: GPT-powered UI Automation for Androidcited by 47
  7. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  8. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  9. 2025Step-GUI Technical Reportcited by 38
  10. 2024LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automationcited by 36
  11. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  12. 2023AXNav: Replaying Accessibility Tests from Natural Languagecited by 31
  13. 2024Dual-View Visual Contextualization for Web Navigationcited by 24
  14. 2024UICrit: Enhancing Automated Design Evaluation with a UICritique Datasetcited by 21
  15. 2025UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesiscited by 20
  16. 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Groundingcited by 20
  17. 2023WebVLN: Vision-and-Language Navigation on Websitescited by 19
  18. 2023A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understandingcited by 18
  19. 2023Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation APIcited by 18
  20. 2024MobileAgent: enhancing mobile control via human-machine interaction and SOP integrationcited by 18
  21. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agentscited by 18
  22. 2025AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMscited by 18
  23. 2025Morae: Proactively Pausing UI Agents for User Choicescited by 18
  24. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  25. 2023Automatic Macro Mining from Interaction Traces at Scalecited by 12
  26. 2024WebQuest: A Benchmark for Multimodal QA on Web Page Sequencescited by 12
  27. 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impactscited by 10
  28. 2024EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Datacited by 10
  29. 2025MapAgent: Trajectory-Constructed Memory-Augmented Planning for Mobile Task Automationcited by 10
  30. 2025OpenApps: Simulating Environment Variations to Measure UI-Agent Reliabilitycited by 10
  31. 2022MUG: Interactive Multimodal Grounding on User Interfacescited by 9
  32. 2025Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmarkcited by 9
  33. 2025Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Questionscited by 8
  34. 2025TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environmentscited by 8
  35. 2025What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilitiescited by 8
  36. 2023UINav: A Practical Approach to Train On-Device Automation Agentscited by 5
  37. 2024Tell Me What's Next: Textual Foresight for Generic UI Representationscited by 5
  38. 2025Agent-Initiated Interaction in Phone UI Automationcited by 4

50 most cited of 61 All 61 (JSON)