mlsanityData to 7 Oct 2026

EliGen: Entity-Level Controlled Image Generation with Regional Attention

arXiv 2501.01097 ↗First posted 2 Jan 2025cs.CVcited by 32

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent advancements in diffusion models have significantly advanced text-to-image generation, yet global text prompts alone remain insufficient for achieving fine-grained control over individual entities within an image. To address this limitation, we present EliGen, a novel framework for Entity-level controlled image Generation. Firstly, we put forward regional attention, a mechanism for diffusion transformers that requires no additional parameters, seamlessly integrating entity prompts and arbitrary-shaped spatial masks. By contributing a high-quality dataset with fine-grained spatial and semantic entity-level annotations, we train EliGen to achieve robust and accurate entity-level manipulation, surpassing existing methods in both spatial precision and image quality. Additionally, we propose an inpainting fusion pipeline, extending its capabilities to multi-entity image inpainting tasks. We further demonstrate its flexibility by integrating it with other open-source models such as IP-Adapter, In-Context LoRA and MLLM, unlocking new creative possibilities. The source code, model, and dataset are published at https://github.com/modelscope/DiffSynth-Studio.git.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 33 of its 40 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

33 of 40 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024In-Context LoRA for Diffusion Transformers31 Oct
  2. 2024Composing Parts for Expressive Object Generation14 Jun
  3. 2024InstanceDiffusion: Instance-level Control for Image Generation5 Feb
  4. 2023Dense Text-to-Image Generation with Attention Modulation24 Aug
  5. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  6. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation30 Mar
  7. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidance25 Feb
  8. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  9. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  11. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  12. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models26 Oct

Other references · 21

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2024Qwen2.5 Technical Reportcited by 4,713
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
9 more
  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  6. 2024YOLOv11: An Overview of the Key Architectural Enhancementscited by 414
  7. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  8. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  9. 2024Learning Multi-dimensional Human Preference for Text-to-Image Generationcited by 102

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 17.

32 papers.

Sort:

By area:

  1. 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 118
  2. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  3. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  4. 2025ContextGen: Contextual Layout Anchoring for Identity-Consistent Multi-Instance Generationcited by 17
  5. 2025Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Controlcited by 10
  6. 2025Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Spacecited by 9
  7. 2025OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlapscited by 9
  8. 2025CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-stepcited by 8
  9. 2025LAMIC: Layout-Aware Multi-Image Composition via Scalability of Multimodal Diffusion Transformercited by 7
  10. 2025AttriCtrl: Fine-Grained Control of Aesthetic Attribute Intensity in Diffusion Modelscited by 5
  11. 2026RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Detailscited by 5
  12. 2025PositionIC: Unified Position and Identity Consistency for Image Customizationcited by 4
20 more
  1. 2025TerraGen: A Unified Multi-Task Layout Generation Framework for Remote Sensing Data Augmentationcited by 4
  2. 2026MetaPoint: Unlocking Precise Spatial Control in Agentic Visual Generationcited by 4
  3. 2025Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Maskingcited by 3
  4. 2026Diffusion Templates: A Unified Plugin Framework for Controllable Diffusioncited by 3
  5. 2026Advancing Aesthetic Image Generation via Composition Transfercited by 2
  6. 2025DreamingComics: A Story Visualization Pipeline via Subject and Layout Customized Generation using Video Modelscited by 1
  7. 2025DreamOmni3: Scribble-based Editing and Generationcited by 1
  8. 2026Shifting the Breaking Point of Flow Matching for Multi-Instance Editingcited by 1
  9. 2026EchoGen: Cycle-Consistent Learning for Unified Layout-Image Generation and Understandingcited by 1
  10. 2026OcclusionFormer: Arranging Z-Order for Layout-Grounded Image Generationcited by 1
  11. 2026Spectral-Progressive Thought Flow for Lightweight Multimodal Reasoningcited by 1
  12. 2026DEIG: Detail-Enhanced Instance Generation with Fine-Grained Semantic Controlcited by 0
  13. 2026AnyPhoto: Multi-Person Identity Preserving Image Generation with ID Adaptive Modulation on Location Canvascited by 0
  14. 2026OneHOI: Unifying Human-Object Interaction Generation and Editingcited by 0
  15. 2026Editing Everything Everywhere All at Oncecited by 0
  16. 2026InstanceControl: Controllable Complex Image Generation without Instance Labelingcited by 0
  17. 2026Appearance Pointers -- Multimodal Region Control of Diffusion Transformerscited by 0
  18. 2026ControlRef: Efficient Layout-Guided Multi-Instance Generation via Anchored 4D-RoPEcited by 0
  19. 2026OccluRank: Controllable Occlusion-Aware Layout-to-Image Generation by Adding Just an Ordinal Rankcited by 0
  20. 2026LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CEIDM: A Controlled Entity and Interaction Diffusion Model for Enhanced Text-to-Image Generation
  2. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models
  3. 2025Precise Parameter Localization for Textual Generation in Diffusion Models
  4. 2023Manipulating Embeddings of Stable Diffusion Prompts
  5. 2024LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation
  6. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  7. 2025UniVG: A Generalist Diffusion Model for Unified Image Generation and Editing
  8. 2023Region-Aware Diffusion for Zero-shot Text-driven Image Editing