# Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answering

arXiv: https://arxiv.org/abs/2511.19768 · HTML: https://mlsanity.com/paper/2511.19768/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation](https://mlsanity.com/paper/2312.03275/) (2023)
- [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024)
- [Context Length Alone Hurts LLM Performance Despite Perfect Retrieval](https://mlsanity.com/paper/2510.05381/) (2025)
- [Words or Vision: Do Vision-Language Models Have Blind Faith in Text?](https://mlsanity.com/paper/2503.02199/) (2025)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning](https://mlsanity.com/paper/2411.17735/) (2024)
- [Explore until Confident: Efficient Exploration for Embodied Question Answering](https://mlsanity.com/paper/2403.15941/) (2024)
- [Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models](https://mlsanity.com/paper/2405.02917/) (2024)
- [GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering](https://mlsanity.com/paper/2412.14480/) (2024)
- [CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space](https://mlsanity.com/paper/2502.12532/) (2025)
- [Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answering](https://mlsanity.com/paper/2503.11117/) (2025)
- [Uncertainty-Aware Evaluation for Vision-Language Models](https://mlsanity.com/paper/2402.14418/) (2024)
- [Identifying and Mitigating Position Bias of Multi-image Vision-Language Models](https://mlsanity.com/paper/2503.13792/) (2025)
- [Reference-free Hallucination Detection for Large Vision-Language Models](https://mlsanity.com/paper/2408.05767/) (2024)
- [Confidence Calibration in Vision-Language-Action Models](https://mlsanity.com/paper/2507.17383/) (2025)
- [Simple Token-Level Confidence Improves Caption Correctness](https://mlsanity.com/paper/2305.07021/) (2023)
- [Map-based Modular Approach for Zero-shot Embodied Question Answering](https://mlsanity.com/paper/2405.16559/) (2024)
- [MaLoRA: Gated Modality LoRA for Key-Space Alignment in Multimodal LLM Fine-Tuning](https://mlsanity.com/paper/2510.26721/) (2025)

## Built on by (most cited first, 2 of 2)

- [HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering](https://mlsanity.com/paper/2602.15513/) (2026)
- [What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents](https://mlsanity.com/paper/2609.39989/) (2026)

## Abstract (arXiv)

Large vision-language models (VLMs) have improved embodied question answering (EQA) agents by providing strong semantic priors for open-vocabulary reasoning. However, when used directly for step-level exploration, VLMs often exhibit frontier oscillations, unstable back-and-forth movements caused by overconfidence and miscalibration, leading to inefficient navigation and degraded answer quality. We propose Prune-Then-Plan, a simple and effective framework that stabilizes exploration through step-level calibration. Instead of trusting raw VLM scores, our method prunes implausible frontier choices using a Holm-Bonferroni inspired pruning procedure and then delegates final decisions to a coverage-based planner. This separation converts overconfident predictions into conservative, interpretable actions by relying on human-level judgments to calibrate the step-level behavior of VLMs. Integrated into the 3D-Mem EQA framework, our approach achieves relative improvements of up to 49% and 33% in visually grounded SPL and LLM-Match metrics respectively over baselines. Overall, our method achieves better scene coverage under equal exploration budgets on both OpenEQA and EXPRESS-Bench datasets.