mlsanityData to 7 Oct 2026

AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant

arXiv 2203.04203 ↗First posted 8 Mar 2022cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

A long-standing goal of intelligent assistants such as AR glasses/robots has been to assist users in affordance-centric real-world scenarios, such as "how can I run the microwave for 1 minute?". However, there is still no clear task definition and suitable benchmarks. In this paper, we define a new task called Affordance-centric Question-driven Task Completion, where the AI assistant should learn from instructional videos to provide step-by-step help in the user's view. To support the task, we constructed AssistQ, a new dataset comprising 531 question-answer samples from 100 newly filmed instructional videos. We also developed a novel Question-to-Actions (Q2A) model to address the AQTC task and validate it on the AssistQ dataset. The results show that our model significantly outperforms several VQA-related baselines while still having large room for improvement. We expect our task and dataset to advance Egocentric AI Assistant's development. Our project page is available at: https://showlab.github.io/assistq/.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 8 Mar 2022.

Builds on

39 of 46 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
27 more
  1. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  2. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  3. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  4. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  5. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  6. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  7. 2016Visual Dialogcited by 377
  8. 2017Embodied Question Answeringcited by 335
  9. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  10. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  11. 2018TarMAC: Targeted Multi-Agent Communicationcited by 201
  12. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  13. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  14. 2018Visual Semantic Navigation using Scene Priorscited by 127
  15. 2019Audio-Visual Scene-Aware Dialogcited by 122
  16. 2018Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learningcited by 91
  17. 2019Multi-Target Embodied Question Answeringcited by 74
  18. 2017Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learningcited by 70
  19. 2021Object-aware Video-language Pre-training for Retrievalcited by 57
  20. 2018Neural Modular Control for Embodied Question Answeringcited by 56
  21. 2018Visual Coreference Resolution in Visual Dialog using Neural Module Networkscited by 54
  22. 2019A Simple Baseline for Audio-Visual Scene-Aware Dialogcited by 52
  23. 2019Two Body Problem: Collaborative Visual Task Completioncited by 45
  24. 2019Learning to Navigate Using Mid-Level Visual Priorscited by 25
  25. 2020Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialogcited by 20
  26. 2020iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answeringcited by 20
  27. 2021Video-Text Pre-training with Learned Regionscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 8; 2024: 5; 2025: 8; 2026: 5.

32 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  3. 2022Egocentric Video-Language Pretrainingcited by 231
  4. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  5. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  6. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  7. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  8. 2023An Outlook into the Future of Egocentric Visioncited by 84
  9. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  10. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  11. 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 43
  12. 2024EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understandingcited by 35
20 more
  1. 2023Affordance Grounding from Demonstration Video to Target Imagecited by 34
  2. 2024ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectivescited by 26
  3. 2024Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackerscited by 24
  4. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 22
  5. 2022Grounded Affordance from Exocentric Viewcited by 21
  6. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  7. 2025EgoBlind: Towards Egocentric Visual Assistance for the Blindcited by 19
  8. 2025EgoWorld: Translating Exocentric View to Egocentric View using Rich Exocentric Observationscited by 8
  9. 2025EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real Worldcited by 7
  10. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  11. 2022Winning the CVPR'2022 AQTC Challenge: A Two-stage Function-centric Approachcited by 3
  12. 2023First Place Solution to the CVPR'2023 AQTC Challenge: A Function-Interaction Centric Approach with Spatiotemporal Visual-Language Alignmentcited by 3
  13. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 2
  14. 2026EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agentcited by 2
  15. 2026DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Groundingcited by 1
  16. 2022The Metaverse Data Deluge: What Can We Do About It?cited by 0
  17. 2022Technical Report for CVPR 2022 LOVEU AQTC Challengecited by 0
  18. 2023A Solution to CVPR'2023 AQTC Challenge: Video Alignment for Multi-Step Inferencecited by 0
  19. 2026EgoErrorVQA: Assess Egocentric Comprehension Capabilities through Procedural Errors for Ego-Agentic AIcited by 0
  20. 2026Companion-style QA Assistance in Ego-Visioncited by 0