mlsanityData to 7 Oct 2026

Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments

arXiv 2104.08560 ↗First posted 17 Apr 2021cs.CL cs.CVcited by 28

Not on the map. Why?

Abstract (arXiv)

In recent years, vision-language research has shifted to study tasks which require more complex reasoning, such as interactive question answering, visual common sense reasoning, and question-answer plausibility prediction. However, the datasets used for these problems fail to capture the complexity of real inputs and multimodal environments, such as ambiguous natural language requests and diverse digital domains. We introduce Mobile app Tasks with Iterative Feedback (MoTIF), a dataset with natural language commands for the greatest number of interactive environments to date. MoTIF is the first to contain natural language requests for interactive environments that are not satisfiable, and we obtain follow-up questions on this subset to enable research on task uncertainty resolution. We perform initial feasibility classification experiments and only reach an F1 score of 37.3, verifying the need for richer vision-language representations and improved architectures to reason about task feasibility.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 17 Apr 2021.

Builds on

14 of 18 references.

References · 14

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  3. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  4. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  5. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  6. 2017Embodied Question Answeringcited by 335
  7. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  8. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  9. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  10. 2021Screen2Vec: Semantic Embedding of GUI Screens and GUI Componentscited by 41
  11. 2018Mapping Natural Language Commands to Web Elementscited by 23
  12. 2020Take the Scenic Route: Improving Generalization in Vision-and-Language Navigationcited by 10
2 more
  1. 2018Visual Dialogue without Vision or Dialoguecited by 6
  2. 2020Determining Question-Answer Plausibility in Crowdsourced Datasets Using Multi-Task Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 13; 2025: 7; 2026: 5.

28 papers.

Sort:

By area:

  1. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  2. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  3. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  4. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  5. 2024On the Effects of Data Scale on UI Control Agentscited by 187
  6. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  7. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  8. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  9. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  10. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 105
  11. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 46
  12. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
16 more
  1. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  2. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
  3. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  4. 2024MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domainscited by 18
  5. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  6. 2025Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigationcited by 13
  7. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  8. 2026SecAgent: Efficient Mobile GUI Agent with Semantic Contextcited by 4
  9. 2025LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmarkcited by 3
  10. 2025MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agentscited by 3
  11. 2024BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Datacited by 2
  12. 2026Do Multi-Agents Dream of Electric Screens? Achieving Perfect Accuracy on AndroidWorld Through Task Decompositioncited by 2
  13. 2026Jev-Mobile: Jev as an Executor for Mobile GUI Agentscited by 2
  14. 2025DigiData: Training and Evaluating General-Purpose Mobile Control Agentscited by 1
  15. 2026GhostUI: Unveiling Hidden Interactions in Mobile UIcited by 1
  16. 2026ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesiscited by 1