mlsanityData to 14 Oct 2026

Scalable 3D Captioning with Pretrained Models

arXiv 2306.07279 ↗First posted 12 Jun 2023cs.CVcited by 203

Not on the map. Why?

Cited by 46 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Cap3D, an automatic approach for generating descriptive text for 3D objects. This approach utilizes pretrained models from image captioning, image-text alignment, and LLM to consolidate captions from multiple views of a 3D asset, completely side-stepping the time-consuming and costly process of manual annotation. We apply Cap3D to the recently introduced large-scale 3D dataset, Objaverse, resulting in 660k 3D-text pairs. Our evaluation, conducted using 41k human annotations from the same dataset, demonstrates that Cap3D surpasses human-authored descriptions in terms of quality, cost, and speed. Through effective prompt engineering, Cap3D rivals human performance in generating geometric descriptions on 17k collected annotations from the ABO dataset. Finally, we finetune Text-to-3D models on Cap3D and human captions, and show Cap3D outperforms; and benchmark the SOTA including Point-E, Shape-E, and DreamFusion.

Key facts. Cited by 203 papers in the map (data through 14 Oct 2026). Builds on 86 of its 116 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

86 of 116 references.

References · 86

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2023GPT-4 Technical Reportcited by 18,396
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  11. 2023Segment Anythingcited by 8,234
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  13. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  14. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  15. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  16. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  17. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  18. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  19. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  20. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  21. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  22. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  23. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  24. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  25. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  26. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  27. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  28. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  29. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  30. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  31. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  32. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  33. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  34. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  35. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  36. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  37. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  38. 2016Self-critical Sequence Training for Image Captioningcited by 547

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 35; 2024: 87; 2025: 48; 2026: 33.

203 papers.

Sort:

By area:

  1. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  2. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
  3. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  4. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 378
  5. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  6. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  7. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  8. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
  9. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 214
  10. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  11. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 189
  12. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
38 more
  1. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  2. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  3. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  4. 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3Dcited by 134
  5. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  6. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 132
  7. 2023Text-to-3D with Classifier Score Distillationcited by 128
  8. 2023Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Modelscited by 123
  9. 2024MV-Adapter: Multi-view Consistent Image Generation Made Easycited by 112
  10. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generationcited by 104
  11. 2024Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Modelscited by 100
  12. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 92
  13. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  14. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priorscited by 74
  15. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  16. 2024Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materialscited by 66
  17. 2024SPAD : Spatially Aware Multiview Diffuserscited by 62
  18. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 62
  19. 2023T3Bench: Benchmarking Current Progress in Text-to-3D Generationcited by 57
  20. 2024VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Modelscited by 56
  21. 2024DreamReward: Text-to-3D Generation with Human Preferencecited by 56
  22. 2024IDOL: Instant Photorealistic 3D Human Creation from a Single Imagecited by 55
  23. 2024Meta 3D TextureGen: Fast and Consistent Texture Generation for 3D Objectscited by 52
  24. 2024GVGEN: Text-to-3D Generation with Volumetric Representationcited by 51
  25. 2024Multi-modal Situated Reasoning in 3D Scenescited by 51
  26. 2023Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusioncited by 50
  27. 2025DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generationcited by 50
  28. 2023Mosaic-SDF for 3D Generative Modelscited by 48
  29. 2023UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generationcited by 47
  30. 2024Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databasescited by 45
  31. 2024One Diffusion to Generate Them Allcited by 41
  32. 2024Vivid-ZOO: Multi-View Video Generation with Diffusion Modelcited by 40
  33. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  34. 2023VolumeDiffusion: Flexible Text-to-3D Generation with Efficient Volumetric Encodercited by 39
  35. 2024LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesiscited by 39
  36. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAEcited by 39
  37. 2023Cascade-Zero123: One Image to Highly Consistent 3D with Self-Prompted Nearby Viewscited by 33
  38. 2024A3VLM: Actionable Articulation-Aware Vision Language Modelcited by 33

50 most cited of 203 All 203 (JSON)