mlsanityData to 7 Oct 2026

Dual-Process Image Generation

arXiv 2506.01955 ↗First posted 2 Jun 2025cs.CV cs.CL cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Prior methods for controlling image generation are limited in their ability to be taught new tasks. In contrast, vision-language models, or VLMs, can learn tasks in-context and produce the correct outputs for a given input. We propose a dual-process distillation scheme that allows feed-forward image generators to learn new tasks from deliberative VLMs. Our scheme uses a VLM to rate the generated images and backpropagates this gradient to update the weights of the image generator. Our general framework enables a wide variety of new control tasks through the same text-and-image based interface. We showcase a handful of applications of this technique for different types of control signals, such as commonsense inferences and visual prompts. With our method, users can implement multimodal controls for properties such as color palette, line weight, horizon position, and relative depth within a matter of minutes. Project page: https://dual-process.github.io.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 61 of its 83 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

61 of 83 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2022Flow Matching for Generative Modelingcited by 5,426
38 more
  1. 2025Qwen2.5-VL Technical Reportcited by 5,418
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  12. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  13. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  14. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  15. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  16. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  17. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  18. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  19. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  20. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  21. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  22. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  23. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  24. 2023Universal Guidance for Diffusion Modelscited by 426
  25. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  26. 2024What matters when building vision-language models?cited by 321
  27. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  28. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  29. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 303
  30. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  31. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  32. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  33. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  34. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  35. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  36. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  37. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  38. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025Learning an Image Editing Model without Image Editing Pairscited by 13
  2. 2026Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuningcited by 8
  3. 2025Product of Experts for Visual Generationcited by 6
  4. 2025Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learningcited by 6
  5. 2025ReasonX: MLLM-Guided Intrinsic Image Decompositioncited by 5
  6. 2026TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Modelscited by 4
  7. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Modelscited by 4
  8. 2026Multimodal Language Models Cannot Spot Spatial Inconsistenciescited by 1
  9. 2026Don't Settle at the Mode! Mitigating Diversity Collapse in Pretrained Flow Models via Feature Self-Guidancecited by 1
  10. 2026In-the-Wild Camouflage Attack on Vehicle Detectors through Controllable Image Editingcited by 0
  11. 2026The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metriccited by 0
  12. 2026X-MULTI: VLM-based Imaging Factor Disentanglement for Factor-Aware Image Synthesiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learners
  2. 2023VIGC: Visual Instruction Generation and Correction
  3. 2026VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models
  4. 2025Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Study
  5. 2026InstanceControl: Controllable Complex Image Generation without Instance Labeling
  6. 2024Instruct-Imagen: Image Generation with Multi-modal Instruction
  7. 2024PUMA: Empowering Unified MLLM with Multi-granular Visual Generation
  8. 2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation