mlsanityData to 7 Oct 2026

IQA: Visual Question Answering in Interactive Environments

arXiv 1712.03316 ↗First posted 9 Dec 2017cs.CVcited by 154

Not on the map. Why?

Abstract (arXiv)

We introduce Interactive Question Answering (IQA), the task of answering questions that require an autonomous agent to interact with a dynamic visual environment. IQA presents the agent with a scene and a question, like: "Are there any apples in the fridge?" The agent must navigate around the scene, acquire visual understanding of scene elements, interact with objects (e.g. open refrigerators) and plan for a series of actions conditioned on the question. Popular reinforcement learning approaches with a single controller perform poorly on IQA owing to the large and diverse state space. We propose the Hierarchical Interactive Memory Network (HIMN), consisting of a factorized set of controllers, allowing the system to operate at multiple levels of temporal abstraction. To evaluate HIMN, we introduce IQUAD V1, a new dataset built upon AI2-THOR, a simulated photo-realistic environment of configurable indoor scenes with interactive objects (code and dataset available at https://github.com/danielgordon10/thor-iqa-cvpr-2018). IQUAD V1 has 75,000 questions, each paired with a unique scene configuration. Our experiments show that our proposed model outperforms popular single controller based methods on IQUAD V1. For sample questions and results, please view our video: https://youtu.be/pXd3C-1jr98

Key facts. Cited by 154 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 42; 2022: 30; 2023: 24; 2024: 29; 2025: 23; 2026: 6.

154 papers.

Sort:

By area:

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  3. 2023LERF: Language Embedded Radiance Fieldscited by 538
  4. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  5. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  6. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  7. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  8. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  9. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  10. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  11. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  12. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
38 more
  1. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  2. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  3. 2022Navigating to Objects in the Real Worldcited by 104
  4. 2021Visual Room Rearrangementcited by 101
  5. 2021Skill Induction and Planning with Latent Languagecited by 99
  6. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  7. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  8. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  9. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  10. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  11. 2023An Outlook into the Future of Egocentric Visioncited by 84
  12. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  13. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  14. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 64
  15. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  16. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  17. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  18. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  19. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  20. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  21. 2024AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectoriescited by 56
  22. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 55
  23. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  24. 2021RobustNav: Towards Benchmarking Robustness in Embodied Navigationcited by 52
  25. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  26. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  27. 2021PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D Worldcited by 47
  28. 2021Learning to Map for Active Semantic Goal Navigationcited by 47
  29. 2022Sound Adversarial Audio-Visual Navigationcited by 46
  30. 2022Retrospectives on the Embodied AI Workshopcited by 46
  31. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?cited by 45
  32. 2024CityNav: A Large-Scale Dataset for Real-World Aerial Navigationcited by 40
  33. 2022Continuous Scene Representations for Embodied AIcited by 39
  34. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  35. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  36. 2022AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistantcited by 32
  37. 2022Episodic Memory Question Answeringcited by 32
  38. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32

50 most cited of 154 All 154 (JSON)