mlsanityData to 14 Oct 2026

Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation

arXiv 2507.08513 ↗First posted 11 Jul 2025cs.GR cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) struggle with accurately capturing camera-object relations, especially for object orientation, camera viewpoint, and camera shots. This stems from the fact that existing MLLMs are trained on images with limited diverse camera-object relations and corresponding textual descriptions. To address this, we propose a synthetic generation pipeline to create large-scale 3D visual instruction datasets. Our framework takes 3D assets as input and uses rendering and diffusion-based image generation models to create photorealistic images preserving precise camera-object relations. Additionally, large language models (LLMs) are used to generate text prompts for guiding visual instruction tuning and controlling image generation. We create Ultimate3D, a dataset of 240K VQAs with precise camera-object annotations, and corresponding benchmark. MLLMs fine-tuned on our proposed dataset outperform commercial models by a large margin, achieving an average accuracy improvement of 33.4% on camera-object relation recognition tasks. Our code, dataset, and benchmark will contribute to broad MLLM applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 70 of its 95 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

70 of 95 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  3. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  4. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  6. 2023VILA: On Pre-training for Visual Language Models12 Dec
  7. 2023What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning2 Nov
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  10. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data20 Aug
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 59

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  12. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  14. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  15. 2016Modeling Context in Referring Expressionscited by 1,063
  16. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  17. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  18. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  19. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  20. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  21. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  22. 2022Kubric: A scalable dataset generatorcited by 381
  23. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  24. 2022Visual Spatial Reasoningcited by 293
  25. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  26. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 233
  27. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  28. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  29. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
  30. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  31. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  32. 2024IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representationcited by 64
  33. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  34. 2024Generative Photography: Scene-Consistent Camera Control for Realistic Text-to-Image Synthesiscited by 26
  35. 2023Diffusion-based Document Layout Generationcited by 21
  36. 2023Generating Images with 3D Annotations Using Diffusion Modelscited by 18
  37. 2024Synth2: Boosting Visual-Language Models with Synthetic Captions and Image Embeddingscited by 18
  38. 2025Generative AI for Cel-Animation: A Surveycited by 17

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data
  2. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts
  3. 20253DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMs
  4. 2023VIGC: Visual Instruction Generation and Correction
  5. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models
  6. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
  7. 2024MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment
  8. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding