mlsanityData to 14 Oct 2026

M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts

arXiv 2312.10763 ↗First posted 17 Dec 2023cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

Recently, 3D understanding has become popular to facilitate autonomous agents to perform further decisionmaking. However, existing 3D datasets and methods are often limited to specific tasks. On the other hand, recent progress in Large Language Models (LLMs) and Multimodal Language Models (MLMs) have demonstrated exceptional general language and imagery tasking performance. Therefore, it is interesting to unlock MLM's potential to be 3D generalist for wider tasks. However, current MLMs' research has been less focused on 3D tasks due to a lack of large-scale 3D instruction-following datasets. In this work, we introduce a comprehensive 3D instructionfollowing dataset called M3DBench, which possesses the following characteristics: 1) It supports general multimodal instructions interleaved with text, images, 3D objects, and other visual prompts. 2) It unifies diverse 3D tasks at both region and scene levels, covering a variety of fundamental abilities in real-world 3D environments. 3) It is a large-scale 3D instruction-following dataset with over 320k instruction-response pairs. Furthermore, we establish a new benchmark for assessing the performance of large models in understanding multi-modal 3D prompts. Extensive experiments demonstrate the effectiveness of our dataset and baseline, supporting general 3D-centric tasks, which can inspire future research.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 54 of its 75 references that are in the map. First posted on arXiv 17 Dec 2023.

Builds on

54 of 75 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  14. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  15. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  16. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  17. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  18. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  19. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  20. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  21. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  22. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  23. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  24. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  25. 20203D Object Detection with Pointformercited by 194
  26. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  27. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  28. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  29. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  30. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  31. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  32. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  33. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
  34. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 99
  35. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
  36. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  37. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  38. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 11; 2026: 4.

32 papers.

Sort:

By area:

  1. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 103
  2. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  3. 2024Multi-modal Situated Reasoning in 3D Scenescited by 51
  4. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  5. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  6. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  7. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  8. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  9. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  10. 2024Quality Assessment in the Era of Large Models: A Surveycited by 26
  11. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 25
  12. 2024GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-trainingcited by 23
20 more
  1. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 23
  2. 2024Space3D-Bench: Spatial 3D Question Answering Benchmarkcited by 17
  3. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  4. 2025Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Modelscited by 15
  5. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
  6. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11
  7. 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMscited by 10
  8. 2024PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Modelcited by 9
  9. 2026Do 3D Large Language Models Really Understand 3D Spatial Relationships?cited by 9
  10. 2024Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Realitycited by 8
  11. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Modelscited by 8
  12. 2025ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Groundingcited by 7
  13. 2026SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Taskscited by 7
  14. 2024LightHouse: A Survey of AGI Hallucinationcited by 4
  15. 2025GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluationcited by 4
  16. 2025JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Roboticscited by 3
  17. 2024Lightweight Model Pre-training via Language Guided Knowledge Distillationcited by 1
  18. 2026Beyond Single Object: Learning 3D Relations with Large Language Modelscited by 1
  19. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomycited by 0
  20. 2026Scaling Diverse Language Generation for 3D Visual Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
  2. 20243DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset
  3. 2025Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation
  4. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  5. 20233D-LLM: Injecting the 3D World into Large Language Models
  6. 20253DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4o
  7. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
  8. 2024Language-Image Models with 3D Understanding