mlsanityData to 7 Oct 2026

CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models

arXiv 2601.21798 ↗First posted 29 Jan 2026cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large Language Models(LLMs) have revolutionized text generation and multimodal perception,but their capabilities in 3D content generation remain underexplored. Existing methods compromise by producing either low-resolution meshes or coarse structural proxies, failing to capture finegrained geometry natively. In this paper, we propose CG-MLLM, a novel Multi-modal Large Language Model (MLLM) capable of 3D captioning and high-resolution 3D generation in a single framework. Leveraging the Mixture-ofTransformer architecture, CG-MLLM decouples disparate modeling needs, where the Token-level Autoregressive (TokenAR) Transformer handles token-level content, and the Block-level Autoregressive (BlockAR) Transformer handles blocklevel content. By integrating a pre-trained visionlanguage backbone with a specialized 3D VAE latent space, CG-MLLM facilitates long-context interactions between standard tokens and spatial blocks within a single integrated architecture. Experimental results show that CG-MLLM significantly outperforms existing MLLMs in generating high-fidelity 3D objects, effectively bringing high-resolution 3D content creation into the mainstream LLM paradigm. Beyond generation, we further observe that learning to produce 3D content transfers back to perception, strengthening the model's image-based 3D understanding.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 76 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

65 of 76 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding2 Jun
  2. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAE25 Nov
  3. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model29 Nov

Other references · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2022Flow Matching for Generative Modelingcited by 5,426
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  10. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  11. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  4. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  5. 2019Root Mean Square Layer Normalizationcited by 1,425
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  8. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,058
  9. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  10. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  12. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  13. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  14. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  15. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  16. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  17. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  18. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  19. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  20. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  21. 2016Language Modeling with Gated Convolutional Networkscited by 688
  22. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  23. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  24. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
  25. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  26. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  27. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  28. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  29. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 254
  30. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  31. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  32. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  33. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  34. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  35. 2025Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Materialcited by 157
  36. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  37. 2024CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refinercited by 120
  38. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformerscited by 2
  2. 2026Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  2. 20263D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
  3. 20233D-LLM: Injecting the 3D World into Large Language Models
  4. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  5. 2025Part-X-MLLM: Part-aware 3D Multimodal Large Language Model
  6. 2024The Revolution of Multimodal Large Language Models: A Survey
  7. 20253DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMs
  8. 2026Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generation