mlsanityData to 14 Oct 2026

3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V

arXiv 2312.09738 ↗First posted 15 Dec 2023cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

In this work, we present a new visual prompting method called 3DAxiesPrompts (3DAP) to unleash the capabilities of GPT-4V in performing 3D spatial tasks. Our investigation reveals that while GPT-4V exhibits proficiency in discerning the position and interrelations of 2D entities through current visual prompting techniques, its abilities in handling 3D spatial tasks have yet to be explored. In our approach, we create a 3D coordinate system tailored to 3D imagery, complete with annotated scale information. By presenting images infused with the 3DAP visual prompt as inputs, we empower GPT-4V to ascertain the spatial positioning information of the given 3D target image with a high degree of precision. Through experiments, We identified three tasks that could be stably completed using the 3DAP method, namely, 2D to 3D Point Reconstruction, 2D to 3D point matching, and 3D Object Detection. We perform experiments on our proposed dataset 3DAP-Data, the results from these experiments validate the efficacy of 3DAP-enhanced GPT-4V inputs, marking a significant stride in 3D spatial task execution.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 59 of its 85 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

59 of 85 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
38 more
  1. 2022Visual Prompt Tuningcited by 1,681
  2. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  3. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  4. 2019How Can We Know What Language Models Know?cited by 959
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  6. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  7. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  8. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  9. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  10. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  11. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  12. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  13. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  14. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  15. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  16. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
  17. 2022Visual Prompting via Image Inpaintingcited by 250
  18. 2023Personalize Segment Anything Model with One Shotcited by 250
  19. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  20. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 202
  21. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  22. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  23. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  24. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  25. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  26. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 154
  27. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 144
  28. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  29. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  30. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 122
  31. 20213D Object Detection for Autonomous Driving: A Surveycited by 106
  32. 2019Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformationcited by 103
  33. 2020Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Imagescited by 79
  34. 2022P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Promptingcited by 74
  35. 20192D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloudcited by 62
  36. 2023Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-trainingcited by 52
  37. 2021P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matchingcited by 42
  38. 2023Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Modelscited by 42

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  2. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  3. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priorscited by 54
  4. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 52
  5. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  6. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  7. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Modelscited by 29
  8. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Modelcited by 21
  9. 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMscited by 10
  10. 2025Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Modelscited by 10
  11. 2026Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Modelscited by 10
  12. 2025Towards Scalable Spatial Intelligence via 2D-to-3D Data Liftingcited by 8
3 more
  1. 20263DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Modelscited by 1
  2. 2024DC3DO: Diffusion Classifier for 3D Objectscited by 0
  3. 2026PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Modelscited by 0