# IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layout

arXiv: https://arxiv.org/abs/2506.01949 · HTML: https://mlsanity.com/paper/2506.01949/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 80 of its 94 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 80)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)

## Built on by (most cited first, 28 of 28)

- [Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization](https://mlsanity.com/paper/2506.15980/) (2025)
- [Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling](https://mlsanity.com/paper/2604.28185/) (2026)
- [MultiMotion: Multi Subject Video Motion Transfer via Video Diffusion Transformer](https://mlsanity.com/paper/2512.07500/) (2025)
- [YOLO-FDA: Integrating Hierarchical Attention and Detail Enhancement for Surface Defect Detection](https://mlsanity.com/paper/2506.21135/) (2025)
- [RefAdGen: High-Fidelity Advertising Image Generation](https://mlsanity.com/paper/2508.11695/) (2025)
- [Normality-Preserving Continual Industrial Anomaly Detection via Orthogonal LoRA Banks](https://mlsanity.com/paper/2606.02042/) (2026)
- [DGE-YOLO: Dual-Branch Gathering and Attention for Accurate UAV Object Detection](https://mlsanity.com/paper/2506.23252/) (2025)
- [DiffFit: Disentangled Garment Warping and Texture Refinement for Virtual Try-On](https://mlsanity.com/paper/2506.23295/) (2025)
- [OptiPrune: Boosting Prompt-Image Consistency with Attention-Guided Noise and Dynamic Token Selection](https://mlsanity.com/paper/2507.00789/) (2025)
- [Conditional Diffusion Model with Anatomical-Dose Dual Constraints for End-to-End Multi-Tumor Dose Prediction](https://mlsanity.com/paper/2508.02043/) (2025)
- [PQ-DAF: Pose-driven Quality-controlled Data Augmentation for Data-scarce Driver Distraction Detection](https://mlsanity.com/paper/2508.10397/) (2025)
- [Animalbooth: multimodal feature enhancement for animal subject personalization](https://mlsanity.com/paper/2509.16702/) (2025)
- [Instruction Guided Multi Object Image Editing with Quantity and Layout Consistency](https://mlsanity.com/paper/2509.24514/) (2025)
- [Span-level Detection of AI-generated Scientific Text via Contrastive Learning and Structural Calibration](https://mlsanity.com/paper/2510.00890/) (2025)
- [AlignVTOFF: Texture-Spatial Feature Alignment for High-Fidelity Virtual Try-Off](https://mlsanity.com/paper/2601.02038/) (2026)
- [Interpretable Logical Anomaly Classification via Constraint Decomposition and Instruction Fine-Tuning](https://mlsanity.com/paper/2602.03530/) (2026)
- [Causal Discovery for Cross-Sectional Data Based on Super-Structure and Divide-and-Conquer](https://mlsanity.com/paper/2602.03914/) (2026)
- [FGAA-FPN: Foreground-Guided Angle-Aware Feature Pyramid Network for Oriented Object Detection](https://mlsanity.com/paper/2602.10710/) (2026)
- [FinSight-Net:A Physics-Aware Decoupled Network with Frequency-Domain Compensation for Underwater Fish Detection in Smart Aquaculture](https://mlsanity.com/paper/2602.19437/) (2026)
- [RSTNet: Enhancing Small-Target Recognition in Noisy SAR Imagery via Robust Feature Learning and Distribution-Aware Regression](https://mlsanity.com/paper/2602.23820/) (2026)
- [RMK RetinaNet: Rotated Multi-Kernel RetinaNet for Robust Oriented Object Detection in Remote Sensing Imagery](https://mlsanity.com/paper/2603.04793/) (2026)
- [Multi-Period Texture Contrast Enhancement for Low-Contrast Wafer Defect Detection and Segmentation](https://mlsanity.com/paper/2603.14282/) (2026)
- [SARES-DEIM: Sparse Mixture-of-Experts Meets DETR for Robust SAR Ship Detection](https://mlsanity.com/paper/2604.04127/) (2026)
- [Beyond Reconstruction: Reconstruction-to-Vector Diffusion for Hyperspectral Anomaly Detection](https://mlsanity.com/paper/2604.11390/) (2026)
- [WILD-SAM: Phase-Aware Expert Adaptation of SAM for Landslide Detection in Wrapped InSAR Interferograms](https://mlsanity.com/paper/2604.14540/) (2026)
- [Boosting Text-to-Image Diffusion Models via Core Token Attention-Based Seed Selection](https://mlsanity.com/paper/2605.19532/) (2026)
- [IB-HFN: Information Bottleneck-Driven SAR-Optical Fusion Network for High-Fidelity Cloud Removal](https://mlsanity.com/paper/2606.09347/) (2026)
- [TRNet: Learning with Topographic Priors for VHR Paddy Rice Mapping](https://mlsanity.com/paper/2608.04154/) (2026)

## Abstract (arXiv)

Despite advances in diffusion-based image editing, manipulating multi-object scenes remains challenging. Existing approaches often achieve semantic changes at the expense of structural consistency, failing to preserve exact object counts and spatial layouts without introducing unintended relocations or background modifications. To address this limitation, we introduce quantity-and-layout-consistent image editing (QL-Edit) to modify object semantics while maintaining the original instance cardinality and spatial layout. We propose IMAGHarmony, a parameter-efficient framework featuring a harmony-aware (HA) module that incorporates perception cues from the reference image into the diffusion process. This enables the model to jointly reason about object semantics, counts, and spatial positions for improved structural consistency. Furthermore, we introduce a preference-guided noise selection (PNS) strategy that identifies favorable initialization conditions, substantially improving generation stability in challenging multi-object scenarios. To support systematic evaluation, we construct HarmonyBench, a benchmark designed to measure semantic editing accuracy and structural consistency under quantity and layout constraints. Extensive experiments demonstrate that IMAGHarmony consistently outperforms existing methods in both structural preservation and semantic accuracy. Notably, our framework is highly efficient, requiring only 200 training images and 10.6M trainable parameters. Code, models, and data are available at urlhttps://github.com/muzishen/IMAGHarmony.