mlsanityData to 7 Oct 2026

Situational Awareness Matters in 3D Vision Language Reasoning

arXiv 2406.07544 ↗First posted 11 Jun 2024cs.CV cs.AI cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Being able to carry out complicated vision language reasoning tasks in 3D space represents a significant milestone in developing household robots and human-centered embodied AI. In this work, we demonstrate that a critical and distinct challenge in 3D vision language reasoning is situational awareness, which incorporates two key components: (1) The autonomous agent grounds its self-location based on a language prompt. (2) The agent answers open-ended questions from the perspective of its calculated position. To address this challenge, we introduce SIG3D, an end-to-end Situation-Grounded model for 3D vision language reasoning. We tokenize the 3D scene into sparse voxel representation and propose a language-grounded situation estimator, followed by a situated question answering module. Experiments on the SQA3D and ScanQA datasets show that SIG3D outperforms state-of-the-art models in situation estimation and question answering by a large margin (e.g., an enhancement of over 30% on situation estimation accuracy). Subsequent analysis corroborates our architectural design choices, explores the distinct functions of visual and textual tokens, and highlights the importance of situational awareness in the domain of 3D question answering.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 64 of its 79 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

64 of 79 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2023Segment Anythingcited by 8,158
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  9. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  10. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  12. 2022Language-driven Semantic Segmentationcited by 711
  13. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  14. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  15. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  16. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  17. 2023LERF: Language Embedded Radiance Fieldscited by 538
  18. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  19. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  20. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  21. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  22. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  23. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  24. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  25. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  26. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  27. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  28. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  29. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  30. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  31. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  32. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  33. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  34. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  35. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  36. 2022Linearly Mapping from Image to Text Spacecited by 139
  37. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  38. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 16; 2026: 10.

32 papers.

Sort:

By area:

  1. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  2. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  3. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  4. 20253D Aware Region Prompted Vision Language Modelcited by 31
  5. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agentscited by 30
  6. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 26
  7. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyondcited by 12
  8. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11
  9. 20253D Question Answering via only 2D Vision-Language Modelscited by 10
  10. 2025ViewSRD: 3D Visual Grounding via Structured Multi-View Decompositioncited by 8
  11. 2025Towards Scalable Spatial Intelligence via 2D-to-3D Data Liftingcited by 8
  12. 2025Empowering Large Language Models with 3D Situation Awarenesscited by 5
20 more
  1. 2025Spatio-Temporal LLM: Reasoning about Environments and Actionscited by 5
  2. 2025CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoningcited by 5
  3. 2024MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentationcited by 4
  4. 2025SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Taskscited by 4
  5. 2025EagleVision: A Dual-Stage Framework with BEV-grounding-based Chain-of-Thought for Spatial Intelligencecited by 3
  6. 2025Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planningcited by 2
  7. 2025Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Modelcited by 2
  8. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  9. 2026SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answeringcited by 2
  10. 2025Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities?cited by 1
  11. 2026What if? Emulative Simulation with World Models for Situated Reasoningcited by 1
  12. 2026Point Cloud as a Foreign Language for Multi-modal Large Language Modelcited by 1
  13. 20263D Segmentation Using Viewpoint-Dependent Spatial Relationshipscited by 1
  14. 2026Grounded 3D-Aware Spatial Vision-Language Modelingcited by 1
  15. 2025Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understandingcited by 0
  16. 2025HMR3D: Hierarchical Multimodal Representation for 3D Scene Understanding with Large Vision-Language Modelcited by 0
  17. 2026OneCanvas: 3D Scene Understanding via Panoramic Reprojectioncited by 0
  18. 2026Agentic Collaborative Cognition for Zero-Shot 3D Understandingcited by 0
  19. 20263DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answeringcited by 0
  20. 2026CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMscited by 0