mlsanityData to 14 Oct 2026

Prompt-Propose-Verify: A Reliable Hand-Object-Interaction Data Generation Framework using Foundational Models

arXiv 2312.15247 ↗First posted 23 Dec 2023cs.CV cs.AIcited by 1

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion models when conditioned on text prompts, generate realistic-looking images with intricate details. But most of these pre-trained models fail to generate accurate images when it comes to human features like hands, teeth, etc. We hypothesize that this inability of diffusion models can be overcome through well-annotated good-quality data. In this paper, we look specifically into improving the hand-object-interaction image generation using diffusion models. We collect a well annotated hand-object interaction synthetic dataset curated using Prompt-Propose-Verify framework and finetune a stable diffusion model on it. We evaluate the image-text dataset on qualitative and quantitative metrics like CLIPScore, ImageReward, Fedility, and alignment and show considerably better performance over the current state-of-the-art benchmarks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 23 of its 27 references that are in the map. First posted on arXiv 23 Dec 2023.

Builds on

23 of 27 references.

References · 23

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
11 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  2. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  3. 2023Effective Data Augmentation With Diffusion Modelscited by 283
  4. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  5. 2018GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classificationcited by 242
  6. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 226
  7. 2022BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotationscited by 99
  8. 2022A Systematic Study of Bias Amplificationcited by 65
  9. 2023AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Posecited by 49
  10. 2023ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attackscited by 28
  11. 2022HandsOff: Labeled Dataset Generation With No Additional Human Annotationscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfercited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data
  2. 2023Affordance Diffusion: Synthesizing Hand-Object Interactions
  3. 2024HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances
  4. 2024Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction
  5. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions
  6. 2026MAD-HOI: Masked Autoregressive Diffusion for Generating Articulated Hand Object Interactions from Text
  7. 2025HOIDiNi: Human-Object Interaction through Diffusion Noise Optimization
  8. 2024Giving a Hand to Diffusion Models: a Two-Stage Approach to Improving Conditional Human Image Generation