mlsanityData to 7 Oct 2026

Explore until Confident: Efficient Exploration for Embodied Question Answering

arXiv 2403.15941 ↗First posted 23 Mar 2024cs.RO cs.AI cs.CVcited by 74

Not on the map. Why?

Abstract (arXiv)

We consider the problem of Embodied Question Answering (EQA), which refers to settings where an embodied agent such as a robot needs to actively explore an environment to gather information until it is confident about the answer to a question. In this work, we leverage the strong semantic reasoning capabilities of large vision-language models (VLMs) to efficiently explore and answer such questions. However, there are two main challenges when using VLMs in EQA: they do not have an internal memory for mapping the scene to be able to plan how to explore over time, and their confidence can be miscalibrated and can cause the robot to prematurely stop exploration or over-explore. We propose a method that first builds a semantic map of the scene based on depth information and via visual prompting of a VLM - leveraging its vast knowledge of relevant regions of the scene for exploration. Next, we use conformal prediction to calibrate the VLM's question answering confidence, allowing the robot to know when to stop exploration - leading to a more calibrated and efficient exploration strategy. To test our framework in simulation, we also contribute a new EQA dataset with diverse, realistic human-robot scenarios and scenes built upon the Habitat-Matterport 3D Research Dataset (HM3D). Both simulated and real robot experiments show our proposed approach improves the performance and efficiency over baselines that do no leverage VLM for exploration or do not calibrate its confidence. Webpage with experiment videos and code: https://explore-eqa.github.io/

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 23 Mar 2024.

Builds on

46 of 63 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  9. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  10. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  11. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  12. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
34 more
  1. 2022Visual Language Maps for Robot Navigationcited by 362
  2. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  3. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  4. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  5. 2017Embodied Question Answeringcited by 335
  6. 20163DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructionscited by 333
  7. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  8. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  9. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  10. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  11. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  12. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  13. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  14. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  15. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  16. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  17. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  18. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  19. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  20. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  21. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  22. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  23. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  24. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  25. 2019Multi-Target Embodied Question Answeringcited by 74
  26. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  27. 2018Neural Modular Control for Embodied Question Answeringcited by 56
  28. 2023How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontierscited by 53
  29. 2021SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistencycited by 47
  30. 2022Adaptive Conformal Prediction for Motion Planning among Dynamic Agentscited by 46
  31. 2022Conformal Prediction for STL Runtime Verificationcited by 34
  32. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  33. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  34. 2022VIBUS: Data-efficient 3D Scene Parsing with VIewpoint Bottleneck and Uncertainty-Spectrum Modelingcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 33; 2026: 25.

74 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  2. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  3. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
  4. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  5. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 55
  6. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  7. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  8. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answeringcited by 37
  9. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  10. 2025CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Spacecited by 31
  11. 2025Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answeringcited by 29
  12. 2025ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmarkcited by 18
38 more
  1. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 18
  2. 2024EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answeringcited by 15
  3. 2024How Generalizable Is My Behavior Cloning Policy? A Statistical Approach to Trustworthy Performance Evaluationcited by 14
  4. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 14
  5. 2024Enhancing Multi-Robot Semantic Navigation Through Multimodal Chain-of-Thought Score Collaborationcited by 14
  6. 2026Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Explorationcited by 13
  7. 2025Memory-Centric Embodied Question Answeringcited by 12
  8. 2025Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answeringcited by 12
  9. 2025UAV-VLN: End-to-End Vision Language guided Navigation for UAVscited by 11
  10. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agentscited by 11
  11. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  12. 2024NoisyEQA: Benchmarking Embodied Question Answering Against Noisy Queriescited by 10
  13. 2025RAVEN: Resilient Aerial Navigation via Open-Set Semantic Memory and Behavior Adaptationcited by 10
  14. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 10
  15. 2025ConformalNL2LTL: Translating Natural Language Instructions into Temporal Logic Formulas with Conformal Correctness Guaranteescited by 9
  16. 2025When Robots Should Say "I Don't Know": Benchmarking Abstention in Embodied Question Answeringcited by 9
  17. 2025WoMAP: World Models For Embodied Open-Vocabulary Object Localizationcited by 8
  18. 2024Wolf: Dense Video Captioning with a World Summarization Frameworkcited by 7
  19. 2024AlignBot: Aligning VLM-powered Customized Task Planning with User Reminders Through Fine-Tuning for Household Robotscited by 7
  20. 2024Can foundation models actively gather information in interactive environments to test hypotheses?cited by 7
  21. 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
  22. 2025From reactive to cognitive: brain-inspired spatial intelligence for embodied agentscited by 7
  23. 2025Uncertainty in Action: Confidence Elicitation in Embodied Agentscited by 5
  24. 2025Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Explorationcited by 5
  25. 2025Toward Ambulatory Vision: Learning Visually-Grounded Active View Selectioncited by 5
  26. 2026GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoningcited by 5
  27. 2025Statistical Methods in Generative AIcited by 4
  28. 2025Human-like Navigation in a World Built for Humanscited by 4
  29. 2025ReEXplore: Improving MLLMs for Embodied Exploration with Contextualized Retrospective Experience Replaycited by 4
  30. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  31. 2024Foundation Models to the Rescue: Deadlock Resolution in Connected Multi-Robot Systemscited by 3
  32. 2024SayComply: Grounding Field Robotic Tasks in Operational Compliance through Retrieval-Based Language Modelscited by 3
  33. 2025Multi-Step Reasoning for Embodied Question Answering via Tool Augmentationcited by 3
  34. 2025Expand Your SCOPE: Semantic Cognition over Potential-Based Exploration for Embodied Visual Navigationcited by 3
  35. 2026HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answeringcited by 3
  36. 2026Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guaranteescited by 3
  37. 2025EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environmentscited by 2
  38. 2025Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answeringcited by 2

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Hierarchical Floorplan-Guided Vision-Language Exploration for Embodied Question Answering ↗
  2. 2026FAST-EQA: Efficient Embodied Question Answering with Global and Local Region Relevancy
  3. 2026Extending Embodied Question Answering from Perception to Decision
  4. 2026ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI
  5. 2026Semantic Flip: Synthetic OOD Generation for Robust Refusal in Embodied Question Answering and Spatial Localization
  6. 2025Embodied Scene Understanding for Vision Language Models via MetaVQA
  7. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigation
  8. 2025NaviTrace: Evaluating Embodied Navigation of Vision-Language Models