mlsanityData to 7 Oct 2026

Counting Guidance for High Fidelity Text-to-Image Synthesis

arXiv 2306.17567 ↗First posted 30 Jun 2023cs.CVcited by 24

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recently, there have been significant improvements in the quality and performance of text-to-image generation, largely due to the impressive results attained by diffusion models. However, text-to-image diffusion models sometimes struggle to create high-fidelity content for the given input prompt. One specific issue is their difficulty in generating the precise number of objects specified in the text prompt. For example, when provided with the prompt "five apples and ten lemons on a table," images generated by diffusion models often contain an incorrect number of objects. In this paper, we present a method to improve diffusion models so that they accurately produce the correct object count based on the input prompt. We adopt a counting network that performs reference-less class-agnostic counting for any given image. We calculate the gradients of the counting network and refine the predicted noise for each step. To address the presence of multiple types of objects in the prompt, we utilize novel attention map guidance to obtain high-quality masks for each object. Finally, we guide the denoising process using the calculated gradients for each object. Through extensive experiments and evaluation, we demonstrate that the proposed method significantly enhances the fidelity of diffusion models with respect to object count. Code is available at https://github.com/furiosa-ai/counting-guidance.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 43 of its 51 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

43 of 51 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Grounded Text-to-Image Synthesis with Attention Refocusing8 Jun
  2. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models23 May
  3. 2023SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models9 May
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  5. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  6. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  7. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
21 more
  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  7. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  8. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  9. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  10. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  11. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  12. 2023Universal Guidance for Diffusion Modelscited by 426
  13. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  14. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  15. 2023Teaching CLIP to Count to Tencited by 194
  16. 2023CLIP-Count: Towards Text-Guided Zero-Shot Object Countingcited by 82
  17. 2022Represent, Compare, and Learn: A Similarity-Aware Framework for Class-Agnostic Countingcited by 80
  18. 2022Few-shot Object Counting with Similarity-Aware Feature Enhancementcited by 62
  19. 2023Zero-shot Object Countingcited by 57
  20. 2022Learning to Count Anything: Reference-less Class-agnostic Counting with Weak Supervisioncited by 41
  21. 2022Exemplar Free Class Agnostic Countingcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 10; 2026: 6.

24 papers.

Sort:

By area:

  1. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 53
  2. 2024Make It Count: Text-to-Image Generation with an Accurate Number of Objectscited by 47
  3. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  4. 2023LoCo: Locally Constrained Training-Free Layout-to-Image Synthesiscited by 21
  5. 2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
  6. 2024Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generationcited by 11
  7. 2025YOLO-Count: Differentiable Object Counting for Text-to-Image Generationcited by 10
  8. 2024Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!cited by 8
  9. 2024Visual Enumeration Remains Challenging for Multimodal Generative AIcited by 5
  10. 2024Detection-Driven Object Count Optimization for Text-to-Image Diffusion Modelscited by 4
  11. 2024Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Modelscited by 4
  12. 2025CountLoop: Training-Free High-Instance Image Generation via Iterative Agent Guidancecited by 4
12 more
  1. 2025CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Explorationcited by 2
  2. 2025Translation of Text Embedding via Delta Vector to Suppress Strongly Entangled Content in Text-to-Image Diffusion Modelscited by 1
  3. 2025CountCluster: Training-Free Object Quantity Guidance with Cross-Attention Map Clustering for Text-to-Image Generationcited by 1
  4. 2025D2D: Detector-to-Differentiable Critic for Improved Numeracy in Text-to-Image Generationcited by 1
  5. 2026When Do Diffusion Models learn to Generate Multiple Objects?cited by 1
  6. 2025ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generationcited by 0
  7. 2025UNCAGE: Contrastive Attention Guidance for Masked Generative Transformers in Text-to-Image Generationcited by 0
  8. 2026ATHENA: Adaptive Test-Time Steering for Improving Count Fidelity in Diffusion Modelscited by 0
  9. 2026Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectificationcited by 0
  10. 2026ELDiff: When Evidential Learning Meets Text-to-Image Diffusioncited by 0
  11. 2026ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generationcited by 0
  12. 2026Learning to Generate Multiple Objects from Dense and Occluded Layoutscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CountDiffusion: Text-to-Image Synthesis with Training-Free Counting-Guidance Diffusion
  2. 2025Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help
  3. 2025Demystifying Numerosity in Diffusion Models -- Limitations and Remedies
  4. 2025CountSteer: Steering Attention for Object Counting in Diffusion Models
  5. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
  6. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis
  7. 2023Semantic Generative Augmentations for Few-Shot Counting
  8. 2024CoCoNO: Attention Contrast-and-Complete for Initial Noise Optimization in Text-to-Image Synthesis