mlsanityData to 7 Oct 2026

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization

arXiv 2502.04370 ↗First posted 5 Feb 2025cs.CL cs.GR cs.LGcited by 27

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Text-to-3D generation automates 3D content creation from textual descriptions, which offers transformative potential across various fields. However, existing methods often struggle to align generated content with human preferences, limiting their applicability and flexibility. To address these limitations, in this paper, we propose DreamDPO, an optimization-based framework that integrates human preferences into the 3D generation process, through direct preference optimization. Practically, DreamDPO first constructs pairwise examples, then compare their alignment with human preferences using reward or large multimodal models, and lastly optimizes the 3D representation with a preference-driven loss function. By leveraging pairwise comparison to reflect preferences, DreamDPO reduces reliance on precise pointwise quality evaluations while enabling fine-grained controllability through preference-guided optimization. Experiments demonstrate that DreamDPO achieves competitive results, and provides higher-quality and more controllable 3D content compared to existing methods. The code and models will be open-sourced.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 92 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

37 of 92 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness27 May
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  4. 2023Aligning Text-to-Image Models using Human Feedback23 Feb

Other references · 33

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  11. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
21 more
  1. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  2. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  3. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  4. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  5. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  7. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  8. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  9. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  10. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  11. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  12. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Modelscited by 154
  13. 2023Text-to-3D with Classifier Score Distillationcited by 127
  14. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidancecited by 109
  15. 2023Noise-Free Score Distillationcited by 99
  16. 2023AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Controlcited by 84
  17. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  18. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  19. 2023HeadSculpt: Crafting 3D Head Avatars with Textcited by 48
  20. 2023Instruct 3D-to-3D: Text Instruction Guided 3D-to-3D conversioncited by 47
  21. 2022Diffusion Guided Domain Adaptation of Image Generatorscited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 10.

27 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundnesscited by 24
  3. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
  4. 2025MagicID: Hybrid Preference Optimization for ID-Consistent and Dynamic-Preserved Video Customizationcited by 17
  5. 2025VINCIE: Unlocking In-context Image Editing from Videocited by 17
  6. 2025Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewardscited by 11
  7. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 11
  8. 2025Zero-1-to-A: Zero-Shot One Image to Animatable Head Avatars Using Video Diffusioncited by 9
  9. 2025Reinforcement Learning for Large Model: A Surveycited by 8
  10. 2025AnchorFlow: Training-Free 3D Editing via Latent Anchor-Aligned Flowscited by 8
  11. 2025DreamCS: Geometry-Aware Text-to-3D Generation with Unpaired 3D Reward Supervisioncited by 7
  12. 2025EvolvingGrasp: Evolutionary Grasp Generation via Efficient Preference Alignmentcited by 6
15 more
  1. 2025Vision-Language Models as Differentiable Semantic and Spatial Rewards for Text-to-3D Generationcited by 6
  2. 2025Reinforced Strategy Optimization for Conversational Recommender Systems via Network-of-Expertscited by 4
  3. 2025Extending Visual Dynamics for Video-to-Music Generationcited by 3
  4. 2026Stroke3D: Lifting 2D strokes into rigged 3D model via latent diffusion modelscited by 2
  5. 2026Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preferencecited by 2
  6. 2026BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenizationcited by 2
  7. 2026A Cross-Model VLM-Judge Protocol for Single-Image 3D Mesh Quality (and Why Cheap Proxies Fall Short)cited by 2
  8. 2025Point Cloud Compression and Objective Quality Assessment: A Surveycited by 1
  9. 2025Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generationcited by 1
  10. 2026iARCS: Iterative Agentic RL for Controllable 3D Scene Generationcited by 1
  11. 2026HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidancecited by 0
  12. 2026Distributed Direct Preference Optimizationcited by 0
  13. 2026Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generationcited by 0
  14. 2026Sculpting NeRF Geometry: Human-Preference Fine-Tuning of a 3D-Aware Face GANcited by 0
  15. 2026Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DreamReward: Text-to-3D Generation with Human Preference
  2. 2025Text-driven 3D Human Generation via Contrastive Preference Optimization
  3. 20253DGen-Bench: Comprehensive Benchmark Suite for 3D Generative Models
  4. 2024A Survey On Text-to-3D Contents Generation In The Wild
  5. 2023T3Bench: Benchmarking Current Progress in Text-to-3D Generation
  6. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation
  7. 2025Dive3D: Diverse Distillation-based Text-to-3D Generation via Score Implicit Matching
  8. 2024Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generation