mlsanityData to 7 Oct 2026

LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models

arXiv 2411.09595 ↗First posted 14 Nov 2024cs.LG cs.AI cs.CLcited by 87

Not on the map. Why?

Abstract (arXiv)

This work explores expanding the capabilities of large language models (LLMs) pretrained on text to generate 3D meshes within a unified model. This offers key advantages of (1) leveraging spatial knowledge already embedded in LLMs, derived from textual sources like 3D tutorials, and (2) enabling conversational 3D generation and mesh understanding. A primary challenge is effectively tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly. To address this, we introduce LLaMA-Mesh, a novel approach that represents the vertex coordinates and face definitions of 3D meshes as plain text, allowing direct integration with LLMs without expanding the vocabulary. We construct a supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate 3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs as required, and (3) understand and interpret 3D meshes. Our work is the first to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge for 3D mesh generation in a text-based format, effectively unifying the 3D and text modalities. LLaMA-Mesh achieves mesh generation quality on par with models trained from scratch while maintaining strong text generation performance.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 38 of its 76 references that are in the map. First posted on arXiv 14 Nov 2024.

Builds on

38 of 76 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2017Decoupled Weight Decay Regularizationcited by 1,722
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  11. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
26 more
  1. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  2. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  3. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  4. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  5. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  6. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
  7. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  8. 2023Text-to-3D using Gaussian Splattingcited by 299
  9. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  10. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  11. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  12. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  13. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 214
  14. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213
  15. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
  16. 2023PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Predictioncited by 165
  17. 2024MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformerscited by 140
  18. 2024Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Imagecited by 132
  19. 2024CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refinercited by 120
  20. 2024MeshAnything V2: Artist-Created Mesh Generation With Adjacent Mesh Tokenizationcited by 102
  21. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 100
  22. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 84
  23. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Modelscited by 54
  24. 2024PivotMesh: Generic 3D Mesh Generation via Pivot Vertices Guidancecited by 47
  25. 2024LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesiscited by 39
  26. 2024The Scene Language: Representing Scenes with Programs, Words, and Embeddingscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 45; 2026: 41.

87 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assetscited by 85
  3. 2025Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modelingcited by 84
  4. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 78
  5. 2025DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learningcited by 62
  6. 2025Ultra3D: Efficient and High-Fidelity 3D Generation with Part Attentioncited by 43
  7. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 34
  8. 2025PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Imagecited by 32
  9. 2025OctGPT: Octree-based Multiscale Autoregressive Models for 3D Shape Generationcited by 27
  10. 2025Generating Physically Stable and Buildable Brick Structures from Textcited by 23
  11. 2025MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Meshcited by 18
  12. 2025Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusioncited by 15
38 more
  1. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  2. 2025iFlame: Interleaving Full and Linear Attention for Efficient Mesh Generationcited by 13
  3. 2025PrimitiveAnything: Human-Crafted 3D Primitive Assembly Generation with Auto-Regressive Transformercited by 12
  4. 2025ARMesh: Autoregressive Mesh Generation via Next-Level-of-Detail Predictioncited by 11
  5. 2026Quantifying the Gap between Understanding and Generation within Unified Multimodal Modelscited by 11
  6. 2026Addressable Memory for Video World Modelscited by 11
  7. 2024Accelerating Video Diffusion Models via Distribution Matchingcited by 10
  8. 2025MonetGPT: Solving Puzzles Enhances MLLMs' Image Retouching Skillscited by 10
  9. 2025CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligencecited by 9
  10. 2026SegviGen: Repurposing 3D Generative Model for Part Segmentationcited by 9
  11. 2025PDT: Point Distribution Transformation with Diffusion Modelscited by 8
  12. 2025MeshRipple: Structured Autoregressive Generation of Artist-Meshescited by 8
  13. 2025Are We Ready for RL in Text-to-3D Generation? A Progressive Investigationcited by 8
  14. 2026ArtLLM: Generating Articulated Assets via 3D LLMcited by 8
  15. 2025DreamCS: Geometry-Aware Text-to-3D Generation with Unpaired 3D Reward Supervisioncited by 7
  16. 2026HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesiscited by 7
  17. 2026PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objectscited by 7
  18. 2025DiMeR: Disentangled Mesh Reconstruction Modelcited by 6
  19. 2025Topology-Preserved Auto-regressive Mesh Generation in the Manner of Weaving Silkcited by 6
  20. 2025Part-X-MLLM: Part-aware 3D Multimodal Large Language Modelcited by 6
  21. 2026LATO.2: Factorized 3D Mesh Generation with Vertex and Topology Flowcited by 6
  22. 2025NURBGen: High-Fidelity Text-to-CAD Generation through LLM-Driven NURBS Modelingcited by 5
  23. 2026LottieGPT: Tokenizing Vector Animation for Autoregressive Generationcited by 5
  24. 2025ARMO: Autoregressive Rigging for Multi-Category Objectscited by 4
  25. 2025LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMscited by 4
  26. 2025SMooGPT: Stylized Motion Generation using Large Language Modelscited by 4
  27. 2025LLMsPark: A Benchmark for Evaluating Large Language Models in Strategic Gaming Contextscited by 4
  28. 2025PartDiffuser: Part-wise 3D Mesh Generation via Discrete Diffusioncited by 4
  29. 2026FACE: A Face-based Autoregressive Representation for High-Fidelity and Efficient Mesh Generationcited by 4
  30. 2026LoST: Level of Semantics Tokenization for 3D Shapescited by 4
  31. 2026MeshTailor: Cutting Seams via Generative Mesh Traversalcited by 4
  32. 2025Autoregressive Generation of Static and Growing Treescited by 3
  33. 2025MetaGen: A DSL, Database, and Benchmark for VLM-Assisted Metamaterial Generationcited by 3
  34. 2025CHARM: Control-point-based 3D Anime Hairstyle Auto-Regressive Modelingcited by 3
  35. 2025ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modelingcited by 3
  36. 2025DragMesh: Interactive 3D Generation Made Easycited by 3
  37. 2026RecGen3D: Reconstruction-Guided 3D Generation in a Shared Canonical Spacecited by 3
  38. 2026MeshFlow: Mesh Generation with Equivariant Flow Matchingcited by 3

50 most cited of 87 All 87 (JSON)