mlsanityData to 14 Oct 2026

Reinforcement Learning from Diffusion Feedback: Q* for Image Search

arXiv 2311.15648 ↗First posted 27 Nov 2023cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large vision-language models are steadily gaining personalization capabilities at the cost of fine-tuning or data augmentation. We present two models for image generation using model-agnostic learning that align semantic priors with generative capabilities. RLDF, or Reinforcement Learning from Diffusion Feedback, is a singular approach for visual imitation through prior-preserving reward function guidance. This employs Q-learning (with standard Q*) for generation and follows a semantic-rewarded trajectory for image search through finite encoding-tailored actions. The second proposed method, noisy diffusion gradient, is optimization driven. At the root of both methods is a special CFG encoding that we propose for continual semantic guidance. Using only a single input image and no text input, RLDF generates high-quality images over varied domains including retail, sports and agriculture showcasing class-consistency and strong visual diversity. Project website is available at https://infernolia.github.io/RLDF.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 79 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

55 of 79 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  12. 2018YOLOv3: An Incremental Improvementcited by 2,659
38 more
  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  2. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  4. 2019Natural Adversarial Examplescited by 1,408
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  6. 2018Demystifying MMD GANscited by 1,293
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  9. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  10. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  11. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  13. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  14. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  15. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  16. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  17. 2023Extracting Training Data from Diffusion Modelscited by 732
  18. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  19. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  20. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  21. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  22. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  23. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  24. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  25. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 424
  26. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  27. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  28. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 327
  29. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  30. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 317
  31. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  32. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  33. 2023Understanding and Mitigating Copying in Diffusion Modelscited by 209
  34. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  35. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 172
  36. 2023Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Modelscited by 153
  37. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  38. 2018Zero-Shot Visual Imitationcited by 117

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Large-scale Reinforcement Learning for Diffusion Models
  2. 2026Step Back to Move Forward: Reflection-Aware Preference Optimization for Visual Generation
  3. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
  4. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models
  5. 2024David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training
  6. 2025AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning
  7. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design
  8. 2025RPO: Fine-Tuning Visual Generative Models via Rich Vision-Language Preferences