# Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering Benchmarks

arXiv: https://arxiv.org/abs/2405.18831 · HTML: https://mlsanity.com/paper/2405.18831/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 29 May 2024.

## Builds on (most cited first, 20 of 20)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)
- [RoboVQA: Multimodal Long-Horizon Reasoning for Robotics](https://mlsanity.com/paper/2311.00899/) (2023)
- [Embodied Question Answering in Photorealistic Environments with Point Cloud Perception](https://mlsanity.com/paper/1904.03461/) (2019)
- [Multi-Target Embodied Question Answering](https://mlsanity.com/paper/1904.04686/) (2019)
- [GroundNLQ @ Ego4D Natural Language Queries Challenge 2023](https://mlsanity.com/paper/2306.15255/) (2023)
- [Evaluating Tool-Augmented Agents in Remote Sensing Platforms](https://mlsanity.com/paper/2405.00709/) (2024)
- [GeckOpt: LLM System Efficiency via Intent-Based Tool Selection](https://mlsanity.com/paper/2404.15804/) (2024)

## Built on by (most cited first, 10 of 10)

- [Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis](https://mlsanity.com/paper/2503.22420/) (2025)
- [Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model](https://mlsanity.com/paper/2408.00754/) (2024)
- [Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering](https://mlsanity.com/paper/2502.00342/) (2025)
- [3D Question Answering via only 2D Vision-Language Models](https://mlsanity.com/paper/2505.22143/) (2025)
- [LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching](https://mlsanity.com/paper/2406.06799/) (2024)
- [3D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answering](https://mlsanity.com/paper/2507.12026/) (2025)
- [Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding](https://mlsanity.com/paper/2608.02980/) (2026)
- [Benchmarking Large Language Models for Image Classification of Marine Mammals](https://mlsanity.com/paper/2410.19848/) (2024)
- [Multi-LLM QA with Embodied Exploration](https://mlsanity.com/paper/2406.10918/) (2024)
- [SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answering](https://mlsanity.com/paper/2511.14567/) (2025)

## Abstract (arXiv)

As interest in "reformulating" the 3D Visual Question Answering (VQA) problem in the context of foundation models grows, it is imperative to assess how these new paradigms influence existing closed-vocabulary datasets. In this case study, we evaluate the zero-shot performance of foundational models (GPT-4 Vision and GPT-4) on well-established 3D VQA benchmarks, namely 3D-VQA and ScanQA. We provide an investigation to contextualize the performance of GPT-based agents relative to traditional modeling approaches. We find that GPT-based agents without any fine-tuning perform on par with the closed vocabulary approaches. Our findings corroborate recent results that "blind" models establish a surprisingly strong baseline in closed-vocabulary settings. We demonstrate that agents benefit significantly from scene-specific vocabulary via in-context textual grounding. By presenting a preliminary comparison with previous baselines, we hope to inform the community's ongoing efforts to refine multi-modal 3D benchmarks.