mlsanityData to 14 Oct 2026

3D Concept Learning and Reasoning from Multi-View Images

arXiv 2303.11327 ↗First posted 20 Mar 2023cs.CV cs.AI cs.LGcited by 60

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Humans are able to accurately reason in 3D by gathering multi-view observations of the surrounding world. Inspired by this insight, we introduce a new large-scale benchmark for 3D multi-view visual question answering (3DMV-VQA). This dataset is collected by an embodied agent actively moving and capturing RGB images in an environment using the Habitat simulator. In total, it consists of approximately 5k scenes, 600k images, paired with 50k questions. We evaluate various state-of-the-art models for visual reasoning on our benchmark and find that they all perform poorly. We suggest that a principled approach for 3D reasoning from multi-view images should be to infer a compact 3D representation of the world from the multi-view images, which is further grounded on open-vocabulary semantic concepts, and then to execute reasoning on these 3D representations. As the first step towards this approach, we propose a novel 3D concept learning and reasoning (3D-CLR) framework that seamlessly combines these components via neural fields, 2D pre-trained vision-language models, and neural reasoning operators. Experimental results suggest that our framework outperforms baseline models by a large margin, but the challenge remains largely unsolved. We further perform an in-depth analysis of the challenges and highlight potential future directions.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 54 of its 75 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

54 of 75 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  7. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 964
  8. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 905
  9. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  10. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  11. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
38 more
  1. 2022Language-driven Semantic Segmentationcited by 717
  2. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 700
  3. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  4. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  5. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 511
  6. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  7. 2021Baking Neural Radiance Fields for Real-Time View Synthesiscited by 468
  8. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  9. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 372
  10. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 370
  11. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  12. 2017Embodied Question Answeringcited by 336
  13. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  14. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  15. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  16. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  17. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 209
  18. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  19. 2017IQA: Visual Question Answering in Interactive Environmentscited by 155
  20. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  21. 2015Order-Embeddings of Images and Languagecited by 113
  22. 2022Habitat-Matterport 3D Semantics Datasetcited by 110
  23. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 108
  24. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  25. 2020Location-aware Graph Convolutional Networks for Video Question Answeringcited by 80
  26. 2019Multi-Target Embodied Question Answeringcited by 74
  27. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  28. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  29. 2022Learning Neural Acoustic Fieldscited by 65
  30. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  31. 2022ComPhy: Compositional Physical Reasoning of Objects and Events from Videoscited by 63
  32. 2019Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Videocited by 60
  33. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 58
  34. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 51
  35. 2017VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentationcited by 50
  36. 2020Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehensioncited by 50
  37. 2021Learning Signal-Agnostic Manifolds of Neural Fieldscited by 45
  38. 20213D Question Answeringcited by 41

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 23; 2025: 17; 2026: 11.

60 papers.

Sort:

By area:

  1. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  2. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  4. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  5. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  6. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  7. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  8. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  9. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 124
  10. 2025Cameras as Relative Positional Encodingcited by 105
  11. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  12. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
38 more
  1. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  2. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 63
  3. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  4. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 49
  5. 2024Physical Property Understanding from Language-Embedded Feature Fieldscited by 48
  6. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  7. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 41
  8. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  9. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 33
  10. 2024Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentationcited by 30
  11. 2025Introducing Visual Perception Token into Multimodal Large Language Modelcited by 26
  12. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  13. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
  14. 2024Neural Fields in Robotics: A Surveycited by 21
  15. 2024NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fieldscited by 20
  16. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 19
  17. 2024Space3D-Bench: Spatial 3D Question Answering Benchmarkcited by 17
  18. 20233D-Aware Visual Question Answering about Parts, Poses and Occlusionscited by 16
  19. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  20. 2024EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answeringcited by 15
  21. 2026RayRoPE: Projective Ray Positional Encoding for Multi-view Attentioncited by 14
  22. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenescited by 11
  23. 2025AerialVG: A Challenging Benchmark for Aerial Visual Grounding by Exploring Positional Relationscited by 10
  24. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  25. 2024PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Modelcited by 9
  26. 2023DAE-Net: Deforming Auto-Encoder for fine-grained shape co-segmentationcited by 7
  27. 2024LLaNA: Large Language and NeRF Assistantcited by 6
  28. 2024LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Controlcited by 6
  29. 2024OpenSU3D: Open World 3D Scene Understanding using Foundation Modelscited by 6
  30. 2024Unified Scene Representation and Reconstruction for 3D Large Language Modelscited by 5
  31. 2025HouseTour: A Virtual Real Estate A(I)gentcited by 5
  32. 2026MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMscited by 4
  33. 2025ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Taskcited by 3
  34. 2025Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning: A Cognitive Science Perspectivecited by 3
  35. 2026VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observationscited by 3
  36. 20263D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decodingcited by 3
  37. 2024Aerial Lifting: Neural Urban Semantic and Building Instance Lifting from Aerial Imagerycited by 2
  38. 2024Beyond Viewpoint: Robust 3D Object Recognition under Arbitrary Views through Joint Multi-Part Representationcited by 1

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA
  2. 2024Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA
  3. 2026CoV: Chain-of-View Prompting for Spatial Reasoning
  4. 2026TriViewBench: Controlled Complexity Scaling for Multi-View Structural Reasoning in MLLMs
  5. 20253D Question Answering via only 2D Vision-Language Models
  6. 2024Language-Image Models with 3D Understanding
  7. 2026Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Views
  8. 2025Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations