mlsanityData to 7 Oct 2026

GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation

arXiv 2401.04092 ↗First posted 8 Jan 2024cs.CVcited by 151

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Despite recent advances in text-to-3D generative methods, there is a notable absence of reliable evaluation metrics. Existing metrics usually focus on a single criterion each, such as how well the asset aligned with the input text. These metrics lack the flexibility to generalize to different evaluation criteria and might not align well with human preferences. Conducting user preference studies is an alternative that offers both adaptability and human-aligned results. User studies, however, can be very expensive to scale. This paper presents an automatic, versatile, and human-aligned evaluation metric for text-to-3D generative models. To this end, we first develop a prompt generator using GPT-4V to generate evaluating prompts, which serve as input to compare text-to-3D models. We further design a method instructing GPT-4V to compare two 3D assets according to user-defined criteria. Finally, we use these pairwise comparison results to assign these models Elo ratings. Experimental results suggest our metric strongly align with human preference across different evaluation criteria.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 61 of its 73 references that are in the map. First posted on arXiv 8 Jan 2024.

Builds on

61 of 73 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  10. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  13. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  14. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  15. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  16. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  17. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  18. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  19. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  20. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  21. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  22. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  23. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  24. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  25. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  26. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  27. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  28. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  29. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  30. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  31. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  32. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  33. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  34. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  35. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  36. 2023How Language Model Hallucinations Can Snowballcited by 310
  37. 2016Revisiting Classifier Two-Sample Testscited by 258
  38. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 56; 2025: 51; 2026: 44.

151 papers.

Sort:

By area:

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  2. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  3. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  4. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  5. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 94
  6. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 91
  7. 2024I-Design: Personalized LLM Interior Designercited by 87
  8. 2025PPTAgent: Generating and Evaluating Presentations Beyond Text-to-Slidescited by 83
  9. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
  10. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priorscited by 72
  11. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  12. 2024ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidancecited by 59
38 more
  1. 2024DreamReward: Text-to-3D Generation with Human Preferencecited by 56
  2. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Modelscited by 55
  3. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agentcited by 53
  4. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  5. 2024Generic 3D Diffusion Adapter Using Controlled Multi-View Editingcited by 46
  6. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  7. 2025Text-to-CadQuery: A New Paradigm for CAD Generation with Scalable Large Model Capabilitiescited by 43
  8. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  9. 2024Rethinking Score Distillation as a Bridge Between Image Distributionscited by 36
  10. 2024PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Modelscited by 36
  11. 2024Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Modelscited by 36
  12. 2024Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Imagescited by 34
  13. 2024Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generationcited by 32
  14. 2024A Comprehensive Survey on 3D Content Generationcited by 28
  15. 2024Segment Anything Model for Road Network Graph Extractioncited by 28
  16. 2024PrEditor3D: Fast and Precise 3D Shape Editingcited by 27
  17. 2024Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioningcited by 26
  18. 2025SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesiscited by 26
  19. 20243DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splattingcited by 25
  20. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 24
  21. 2024CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesiscited by 23
  22. 2024Text2CAD: Generating Sequential CAD Models from Beginner-to-Expert Level Text Promptscited by 22
  23. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?cited by 20
  24. 2024GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMscited by 20
  25. 2024Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabricationcited by 19
  26. 2024Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearningcited by 19
  27. 20243D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generationcited by 19
  28. 2024BlenderAlchemy: Editing 3D Graphics with Vision-Language Modelscited by 18
  29. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  30. 2025Infinite Mobility: Scalable High-Fidelity Synthesis of Articulated Objects via Procedural Generationcited by 17
  31. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scalecited by 17
  32. 2025Large Language Models for Computer-Aided Design: A Surveycited by 16
  33. 2024Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generationcited by 15
  34. 2025Eval3D: Interpretable and Fine-grained Evaluation for 3D Generationcited by 15
  35. 2024Text-to-3D Shape Generationcited by 13
  36. 2024A Survey On Text-to-3D Contents Generation In The Wildcited by 13
  37. 2024LangXAI: Integrating Large Vision Models for Generating Textual Explanations to Enhance Explainability in Visual Perception Taskscited by 12
  38. 2024MvDrag3D: Drag-based Creative 3D Editing via Multi-view Generation-Reconstruction Priorscited by 12

50 most cited of 151 All 151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
  2. 2025Text-driven 3D Human Generation via Contrastive Preference Optimization
  3. 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation
  4. 2025Sel3DCraft: Interactive Visual Prompts for User-Friendly Text-to-3D Generation
  5. 2023Debiasing Scores and Prompts of 2D Diffusion for View-consistent Text-to-3D Generation
  6. 2025Evaluating the Evaluators: Metrics for Compositional Text-to-Image Generation
  7. 2024TypeScore: A Text Fidelity Metric for Text-to-Image Generative Models
  8. 2024A Quantitative Evaluation of Score Distillation Sampling Based Text-to-3D