# Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments

arXiv: https://arxiv.org/abs/2104.08560 · HTML: https://mlsanity.com/paper/2104.08560/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 18 references that are in the map.
- First posted on arXiv 17 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [IQA: Visual Question Answering in Interactive Environments](https://mlsanity.com/paper/1712.03316/) (2017)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [Mapping Natural Language Commands to Web Elements](https://mlsanity.com/paper/1808.09132/) (2018)
- [Take the Scenic Route: Improving Generalization in Vision-and-Language Navigation](https://mlsanity.com/paper/2003.14269/) (2020)
- [Visual Dialogue without Vision or Dialogue](https://mlsanity.com/paper/1812.06417/) (2018)
- [Determining Question-Answer Plausibility in Crowdsourced Datasets Using Multi-Task Learning](https://mlsanity.com/paper/2011.04883/) (2020)

## Built on by (most cited first, 28 of 28)

- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [On the Effects of Data Scale on UI Control Agents](https://mlsanity.com/paper/2406.03679/) (2024)
- [GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices](https://mlsanity.com/paper/2406.08451/) (2024)
- [Android in the Zoo: Chain-of-Action-Thought for GUI Agents](https://mlsanity.com/paper/2403.02713/) (2024)
- [OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web](https://mlsanity.com/paper/2402.17553/) (2024)
- [GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation](https://mlsanity.com/paper/2311.07562/) (2023)
- [AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents](https://mlsanity.com/paper/2407.17490/) (2024)
- [AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents](https://mlsanity.com/paper/2410.24024/) (2024)
- [MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding](https://mlsanity.com/paper/2409.14818/) (2024)
- [LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects](https://mlsanity.com/paper/2504.19838/) (2025)
- [Foundations and Recent Trends in Multimodal Mobile Agents: A Survey](https://mlsanity.com/paper/2411.02006/) (2024)
- [Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale](https://mlsanity.com/paper/2201.04100/) (2022)
- [Falcon-UI: Understanding GUI Before Following User Instructions](https://mlsanity.com/paper/2412.09362/) (2024)
- [MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains](https://mlsanity.com/paper/2407.18961/) (2024)
- [Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agents](https://mlsanity.com/paper/2505.11891/) (2025)
- [Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey](https://mlsanity.com/paper/2506.11102/) (2025)
- [Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation](https://mlsanity.com/paper/2506.18158/) (2025)
- [LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark](https://mlsanity.com/paper/2504.13805/) (2025)
- [SecAgent: Efficient Mobile GUI Agent with Semantic Context](https://mlsanity.com/paper/2603.08533/) (2026)
- [MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents](https://mlsanity.com/paper/2512.12634/) (2025)
- [Jev-Mobile: Jev as an Executor for Mobile GUI Agents](https://mlsanity.com/paper/2609.30186/) (2026)
- [BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data](https://mlsanity.com/paper/2410.00773/) (2024)
- [Do Multi-Agents Dream of Electric Screens? Achieving Perfect Accuracy on AndroidWorld Through Task Decomposition](https://mlsanity.com/paper/2602.07787/) (2026)
- [ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis](https://mlsanity.com/paper/2605.25160/) (2026)
- [DigiData: Training and Evaluating General-Purpose Mobile Control Agents](https://mlsanity.com/paper/2511.07413/) (2025)
- [GhostUI: Unveiling Hidden Interactions in Mobile UI](https://mlsanity.com/paper/2601.19258/) (2026)

## Abstract (arXiv)

In recent years, vision-language research has shifted to study tasks which require more complex reasoning, such as interactive question answering, visual common sense reasoning, and question-answer plausibility prediction. However, the datasets used for these problems fail to capture the complexity of real inputs and multimodal environments, such as ambiguous natural language requests and diverse digital domains. We introduce Mobile app Tasks with Iterative Feedback (MoTIF), a dataset with natural language commands for the greatest number of interactive environments to date. MoTIF is the first to contain natural language requests for interactive environments that are not satisfiable, and we obtain follow-up questions on this subset to enable research on task uncertainty resolution. We perform initial feasibility classification experiments and only reach an F1 score of 37.3, verifying the need for richer vision-language representations and improved architectures to reason about task feasibility.