mlsanityData to 7 Oct 2026

ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots

arXiv 2209.08199 ↗First posted 16 Sep 2022cs.CL cs.CV cs.HCcited by 65

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce ScreenQA, a novel benchmarking dataset designed to advance screen content understanding through question answering. The existing screen datasets are focused either on low-level structural and component understanding, or on a much higher-level composite task such as navigation and task completion for autonomous agents. ScreenQA attempts to bridge this gap. By annotating 86k question-answer pairs over the RICO dataset, we aim to benchmark the screen reading comprehension capacity, thereby laying the foundation for vision-based automation over screenshots. Our annotations encompass full answers, short answer phrases, and corresponding UI contents with bounding boxes, enabling four subtasks to address various application scenarios. We evaluate the dataset's efficacy using both open-weight and proprietary models in zero-shot, fine-tuned, and transfer learning settings. We further demonstrate positive transfer to web applications, highlighting its potential beyond mobile applications.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 32 of its 83 references that are in the map. First posted on arXiv 16 Sep 2022.

Builds on

32 of 83 references.

References · 32

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  12. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
20 more
  1. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  2. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  3. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  4. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  5. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  6. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  7. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  8. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  9. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  10. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  11. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  12. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  13. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  14. 2019Towards Unconstrained End-to-End Text Spottingcited by 57
  15. 2019TWEETQA: A Social Media Focused Question Answering Datasetcited by 55
  16. 2019LEAF-QA: Locate, Encode & Attend for Figure Question Answeringcited by 53
  17. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
  18. 2020Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learningcited by 25
  19. 2021Wireframe-Based UI Design Search Through Image Autoencodercited by 17
  20. 2021Multimodal Icon Annotation For Mobile Applicationscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 18; 2025: 30; 2026: 14.

65 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  4. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  5. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  6. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  7. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  8. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  9. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  10. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  11. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  12. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
38 more
  1. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  2. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  3. 2025FineVision: Open Data Is All You Needcited by 51
  4. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 46
  5. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  6. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  7. 2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28
  8. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25
  9. 2025Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Modelscited by 24
  10. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  11. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  12. 2023WebVLN: Vision-and-Language Navigation on Websitescited by 19
  13. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  14. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  15. 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
  16. 2024POINTS: Improving Your Vision-language Model with Affordable Strategiescited by 17
  17. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  18. 2025NVIDIA Nemotron Nano V2 VLcited by 15
  19. 2025ProBench: Benchmarking GUI Agents with Accurate Process Informationcited by 14
  20. 2025ReachAgent: Enhancing Mobile Agent via Page Reaching and Operationcited by 13
  21. 2024WebQuest: A Benchmark for Multimodal QA on Web Page Sequencescited by 12
  22. 2024TinyClick: Single-Turn Agent for Empowering GUI Automationcited by 12
  23. 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
  24. 2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling Systemcited by 11
  25. 2025History-Aware Reasoning for GUI Agentscited by 11
  26. 2025PPTArena: A Benchmark for PowerPoint Editingcited by 9
  27. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  28. 2024On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoningcited by 7
  29. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoningcited by 7
  30. 2025KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contextscited by 6
  31. 2026Generalization in Online Reinforcement Learning for Mobile Agentscited by 6
  32. 2025Surfer 2: The Next Generation of Cross-Platform Computer Use Agentscited by 5
  33. 2025AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Modelcited by 4
  34. 2026MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Refluxcited by 4
  35. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Modelcited by 3
  36. 2025PP-DocBee: Improving Multimodal Document Understanding Through a Bag of Trickscited by 3
  37. 2026POINTS-GUI-G: GUI-Grounding Journeycited by 3
  38. 2026Scaling GUI Agents with Visual State Transitionscited by 3

50 most cited of 65 All 65 (JSON)