mlsanityData to 14 Oct 2026

VerbDiff: Text-Only Diffusion Models with Enhanced Interaction Awareness

arXiv 2503.16406 ↗First posted 20 Mar 2025cs.GR cs.CV cs.MMcited by 7

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent large-scale text-to-image diffusion models generate photorealistic images but often struggle to accurately depict interactions between humans and objects due to their limited ability to differentiate various interaction words. In this work, we propose VerbDiff to address the challenge of capturing nuanced interactions within text-to-image diffusion models. VerbDiff is a novel text-to-image generation model that weakens the bias between interaction words and objects, enhancing the understanding of interactions. Specifically, we disentangle various interaction words from frequency-based anchor words and leverage localized interaction regions from generated images to help the model better capture semantics in distinctive words without extra conditions. Our approach enables the model to accurately understand the intended interaction between humans and objects, producing high-quality images with accurate interactions aligned with specified verbs. Extensive experiments on the HICO-DET dataset demonstrate the effectiveness of our method compared to previous approaches.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 28 of its 36 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

28 of 36 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidance24 May
  2. 2024Relation Rectification in Diffusion Model29 Mar
  3. 2024Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing6 Mar
  4. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts16 Oct
  5. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
  6. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models23 May
  7. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation30 Mar
  8. 2023ReVersion: Diffusion-Based Relation Inversion from Images23 Mar
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  10. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan

Other references · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  9. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  10. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  11. 2014Deep metric learning using Triplet networkcited by 422
  12. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
6 more
  1. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  2. 2017Learning to Detect Human-Object Interactionscited by 240
  3. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  4. 2022Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Trainingcited by 56
  5. 2024ReCorD: Reasoning and Correcting Diffusion for HOI Generationcited by 6
  6. 2023Focusing on what to decode and what to train: SOV Decoding with Specific Target Guided DeNoising and Vision Language Advisorcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025SIDA: Synthetic Image Driven Zero-shot Domain Adaptationcited by 4
  2. 2025Sali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioningcited by 4
  3. 2025SynC: Synthetic Image Caption Dataset Refinement with One-to-many Mapping for Zero-shot Image Captioningcited by 3
  4. 2026SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioningcited by 1
  5. 2026Composing People Together: Iterative Pose-Image Generation for Multi-Person Interaction Scenescited by 1
  6. 2026ADAPT: Attention Driven Adaptive Prompt Scheduling and InTerpolating Orthogonal Complements for Rare Concepts Generationcited by 0
  7. 2026OneHOI: Unifying Human-Object Interaction Generation and Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models
  2. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decomposition
  3. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models
  4. 2023De-Diffusion Makes Text a Strong Cross-Modal Interface
  5. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models
  6. 2023DiffBlender: Composable and Versatile Multimodal Text-to-Image Diffusion Models
  7. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
  8. 2023The Hidden Language of Diffusion Models