# Counting Guidance for High Fidelity Text-to-Image Synthesis

arXiv: https://arxiv.org/abs/2306.17567 · HTML: https://mlsanity.com/paper/2306.17567/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 43 of its 51 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16381/) (2023)

## Built on by (most cited first, 24 of 24)

- [LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts](https://mlsanity.com/paper/2310.10640/) (2023)
- [Make It Count: Text-to-Image Generation with an Accurate Number of Objects](https://mlsanity.com/paper/2406.10210/) (2024)
- [IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layout](https://mlsanity.com/paper/2506.01949/) (2025)
- [LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis](https://mlsanity.com/paper/2311.12342/) (2023)
- [Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Models](https://mlsanity.com/paper/2506.01320/) (2025)
- [Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generation](https://mlsanity.com/paper/2411.18301/) (2024)
- [YOLO-Count: Differentiable Object Counting for Text-to-Image Generation](https://mlsanity.com/paper/2508.00728/) (2025)
- [Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!](https://mlsanity.com/paper/2410.20972/) (2024)
- [Visual Enumeration Remains Challenging for Multimodal Generative AI](https://mlsanity.com/paper/2402.03328/) (2024)
- [Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models](https://mlsanity.com/paper/2410.22775/) (2024)
- [Detection-Driven Object Count Optimization for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2408.11721/) (2024)
- [CountLoop: Training-Free High-Instance Image Generation via Iterative Agent Guidance](https://mlsanity.com/paper/2508.16644/) (2025)
- [CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration](https://mlsanity.com/paper/2509.17458/) (2025)
- [Translation of Text Embedding via Delta Vector to Suppress Strongly Entangled Content in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2508.10407/) (2025)
- [CountCluster: Training-Free Object Quantity Guidance with Cross-Attention Map Clustering for Text-to-Image Generation](https://mlsanity.com/paper/2508.10710/) (2025)
- [D2D: Detector-to-Differentiable Critic for Improved Numeracy in Text-to-Image Generation](https://mlsanity.com/paper/2510.19278/) (2025)
- [When Do Diffusion Models learn to Generate Multiple Objects?](https://mlsanity.com/paper/2605.00273/) (2026)
- [ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation](https://mlsanity.com/paper/2505.20935/) (2025)
- [UNCAGE: Contrastive Attention Guidance for Masked Generative Transformers in Text-to-Image Generation](https://mlsanity.com/paper/2508.05399/) (2025)
- [ATHENA: Adaptive Test-Time Steering for Improving Count Fidelity in Diffusion Models](https://mlsanity.com/paper/2603.19676/) (2026)
- [Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectification](https://mlsanity.com/paper/2604.16993/) (2026)
- [ELDiff: When Evidential Learning Meets Text-to-Image Diffusion](https://mlsanity.com/paper/2606.20924/) (2026)
- [ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation](https://mlsanity.com/paper/2606.23835/) (2026)
- [Learning to Generate Multiple Objects from Dense and Occluded Layouts](https://mlsanity.com/paper/2607.03488/) (2026)

## Abstract (arXiv)

Recently, there have been significant improvements in the quality and performance of text-to-image generation, largely due to the impressive results attained by diffusion models. However, text-to-image diffusion models sometimes struggle to create high-fidelity content for the given input prompt. One specific issue is their difficulty in generating the precise number of objects specified in the text prompt. For example, when provided with the prompt "five apples and ten lemons on a table," images generated by diffusion models often contain an incorrect number of objects. In this paper, we present a method to improve diffusion models so that they accurately produce the correct object count based on the input prompt. We adopt a counting network that performs reference-less class-agnostic counting for any given image. We calculate the gradients of the counting network and refine the predicted noise for each step. To address the presence of multiple types of objects in the prompt, we utilize novel attention map guidance to obtain high-quality masks for each object. Finally, we guide the denoising process using the calculated gradients for each object. Through extensive experiments and evaluation, we demonstrate that the proposed method significantly enhances the fidelity of diffusion models with respect to object count. Code is available at https://github.com/furiosa-ai/counting-guidance.