mlsanityData to 7 Oct 2026

Readout Guidance: Learning Control from Diffusion Features

arXiv 2312.02150 ↗First posted 4 Dec 2023cs.CVcited by 52

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple images, such as correspondence and appearance similarity. Furthermore, by comparing the readout estimates to a user-defined target, and back-propagating the gradient through the readout head, these estimates can be used to guide the sampling process. Compared to prior methods for conditional generation, Readout Guidance requires significantly fewer added parameters and training samples, and offers a convenient and simple recipe for reproducing different forms of conditional control under a single framework, with a single architecture and sampling procedure. We showcase these benefits in the applications of drag-based manipulation, identity-consistent generation, and spatially aligned control. Project page: https://readout-guidance.github.io.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 52 of its 67 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

52 of 67 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  2. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models11 Aug
  3. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models5 Jul
  4. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing26 Jun
  5. 2023Emergent Correspondence from Image Diffusion6 Jun
  6. 2023Unsupervised Semantic Correspondence Using Stable Diffusion24 May
  7. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence24 May
  8. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence23 May
  9. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models23 May
  10. 2023Unleashing Text-to-Image Diffusion Models for Visual Perception3 Mar
  11. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
4 more
  1. 2022Shape-Guided Diffusion with Inside-Outside Attention1 Dec
  2. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
24 more
  1. 2021Vision Transformers for Dense Predictioncited by 1,789
  2. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  3. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  4. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  6. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  7. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  8. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  9. 2015Holistically-Nested Edge Detectioncited by 590
  10. 2023CoTracker: It is Better to Track Togethercited by 516
  11. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  12. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  13. 2023Universal Guidance for Diffusion Modelscited by 426
  14. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  15. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  16. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  17. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  18. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  19. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 303
  20. 2023Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldcited by 218
  21. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  22. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 107
  23. 2023RealFill: Reference-Driven Generation for Authentic Image Completioncited by 58
  24. 2022DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 27; 2026: 12.

52 papers.

Sort:

By area:

  1. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  2. 2024Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidancecited by 108
  3. 2024Video Diffusion Models are Training-free Motion Interpreter and Controllercited by 60
  4. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  5. 2024Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimatorscited by 39
  6. 2024Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videoscited by 38
  7. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  8. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  9. 2024Pathways on the Image Manifold: Image Editing via Video Generationcited by 27
  10. 2025Drag Your Gaussian: Effective Drag-Based Editing with Score Distillation for 3D Gaussian Splattingcited by 21
  11. 2025ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motionscited by 19
  12. 2024LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videoscited by 18
38 more
  1. 2025SILO: Solving Inverse Problems with Latent Operatorscited by 18
  2. 2024A Simple Approach to Unifying Diffusion-based Conditional Generationcited by 13
  3. 2025FramePainter: Endowing Interactive Image Editing with Video Diffusion Priorscited by 13
  4. 2025Training-free Geometric Image Editing on Diffusion Modelscited by 13
  5. 2025Dual-Process Image Generationcited by 12
  6. 2025Compass Control: Multi Object Orientation Control for Text-to-Image Generationcited by 11
  7. 2025JointDiT: Enhancing RGB-Depth Joint Modeling with Diffusion Transformerscited by 11
  8. 2025Be Decisive: Noise-Induced Layouts for Multi-Subject Generationcited by 10
  9. 2025PractiLight: Practical Light Control Using Foundational Diffusion Modelscited by 7
  10. 2025Inpaint4Drag: Repurposing Inpainting Models for Drag-Based Image Editing via Bidirectional Warpingcited by 6
  11. 2025Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Modelscited by 5
  12. 2025Cora: Correspondence-aware image editing using few step diffusioncited by 5
  13. 2025CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generationcited by 4
  14. 2025SONIC: Spectral Optimization of Noise for Inpainting with Consistencycited by 4
  15. 2026Extend3D: Town-Scale 3D Generationcited by 4
  16. 2025MeSS: City Mesh-Guided Outdoor Scene Generation with Cross-View Consistent Diffusioncited by 3
  17. 2024GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Designcited by 2
  18. 2025DepthSync: Diffusion Guidance-Based Depth Synchronization for Scale- and Geometry-Consistent Video Depth Estimationcited by 2
  19. 2025Neural USD: An object-centric framework for iterative editing and controlcited by 2
  20. 2024Generating Intermediate Representations for Compositional Text-To-Image Generationcited by 1
  21. 2025MeshFleet: Filtered and Annotated 3D Vehicle Dataset for Domain Specific Generative Modelingcited by 1
  22. 2025Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedbackcited by 1
  23. 2025DirectDrag: High-Fidelity, Mask-Free, Prompt-Free Drag-based Image Editing via Readout-Guided Feature Alignmentcited by 1
  24. 2025RealDrag: The First Dragging Benchmark with Real Target Imagecited by 1
  25. 2026Low-Resource Guidance for Controllable Latent Audio Diffusioncited by 1
  26. 2026ChArtist: Generating Pictorial Charts with Unified Spatial and Subject Controlcited by 1
  27. 2025Guiding Diffusion with Deep Geometric Moments: Balancing Fidelity and Variationcited by 0
  28. 2025Diffusion-Classifier Synergy: Reward-Aligned Learning via Mutual Boosting Loop for FSCILcited by 0
  29. 2025FreeFuse: Multi-Subject LoRA Fusion via Adaptive Token-Level Routing at Test Timecited by 0
  30. 2025DiffusionBrowser: Interactive Diffusion Previews via Multi-Branch Decoderscited by 0
  31. 2025An evaluation of SVBRDF Prediction from Generative Image Models for Appearance Modeling of 3D Scenescited by 0
  32. 2026DynaDrag: Dynamic Drag-Style Image Editing by Motion Predictioncited by 0
  33. 2026Latent Structure Emergence in Diffusion Models via Confidence-Based Filteringcited by 0
  34. 2026FlowBender: Feedback-Aware Training for Self-Correcting Conditional Flowscited by 0
  35. 2026In-context Region-based Drag: Drag Any Region to Any Shapecited by 0
  36. 2026OPERA: Object Perception Enhances Single-view 3D Reconstructioncited by 0
  37. 2026Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generationcited by 0
  38. 2026ReaDiT Guidance: Control for Image and Video Generation using Diffusion Transformer Featurescited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025REG: Rectified Gradient Guidance for Conditional Diffusion Models
  2. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance
  3. 2024Enhancing Image Layout Control with Loss-Guided Diffusion Models
  4. 2024ECNet: Effective Controllable Text-to-Image Diffusion Models
  5. 2025Entropy Rectifying Guidance for Diffusion and Flow Models
  6. 2025Feedback Guidance of Diffusion Models
  7. 2024Plug-and-Play Diffusion Distillation
  8. 2023Improving Diffusion-based Image Translation using Asymmetric Gradient Guidance