mlsanityData to 14 Oct 2026

Generating Context-Aware Natural Answers for Questions in 3D Scenes

arXiv 2310.19516 ↗First posted 30 Oct 2023cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

3D question answering is a young field in 3D vision-language that is yet to be explored. Previous methods are limited to a pre-defined answer space and cannot generate answers naturally. In this work, we pivot the question answering task to a sequence generation task to generate free-form natural answers for questions in 3D scenes (Gen3DQA). To this end, we optimize our model directly on the language rewards to secure the global sentence semantics. Here, we also adapt a pragmatic language understanding reward to further improve the sentence quality. Our method sets a new SOTA on the ScanQA benchmark (CIDEr score 72.22/66.57 on the test sets).

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

47 of 59 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2016Modeling Context in Referring Expressionscited by 1,063
35 more
  1. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  2. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  3. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  4. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  5. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  6. 2016Self-critical Sequence Training for Image Captioningcited by 547
  7. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  8. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  9. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  10. 2022Multimodal Learning with Transformers: A Surveycited by 400
  11. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  12. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  13. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  14. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  15. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  16. 2016Segmentation from Natural Language Expressionscited by 257
  17. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  18. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 200
  19. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  20. 2015Natural Language Object Retrievalcited by 124
  21. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  22. 2017Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challengecited by 76
  23. 2016A Joint Speaker-Listener-Reinforcer Model for Referring Expressionscited by 74
  24. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  25. 2021Video as Conditional Graph Hierarchy for Multi-Granular Question Answeringcited by 68
  26. 2022Invariant Grounding for Video Question Answeringcited by 68
  27. 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
  28. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  29. 2020InterBERT: Vision-and-Language Interaction for Multi-modal Pretrainingcited by 48
  30. 20213D Question Answeringcited by 41
  31. 2016Dense Captioning with Joint Inference and Visual Contextcited by 40
  32. 2019Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioningcited by 35
  33. 2020A Better Variant of Self-Critical Sequence Trainingcited by 23
  34. 2019Visual Question Answering using Deep Learning: A Survey and Performance Analysiscited by 19
  35. 2021Video Question Answering with Phrases via Semantic Rolescited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 3.

5 papers.

  1. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  2. 2024Reconstructing Animals and the Wildcited by 7
  3. 20253D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answeringcited by 5
  4. 2026BrickNet: Graph-Backed Generative Brick Assemblycited by 5
  5. 2026GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Modelscited by 3