mlsanityData to 14 Oct 2026

GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos

arXiv 2312.07322 ↗First posted 12 Dec 2023cs.CVcited by 40

Not on the map. Why?

Abstract (arXiv)

We address the task of generating temporally consistent and physically plausible images of actions and object state transformations. Given an input image and a text prompt describing the targeted transformation, our generated images preserve the environment and transform objects in the initial image. Our contributions are threefold. First, we leverage a large body of instructional videos and automatically mine a dataset of triplets of consecutive frames corresponding to initial object states, actions, and resulting object transformations. Second, equipped with this data, we develop and train a conditioned diffusion model dubbed GenHowTo. Third, we evaluate GenHowTo on a variety of objects and actions and show superior performance compared to existing methods. In particular, we introduce a quantitative evaluation where GenHowTo achieves 88% and 74% on seen and unseen interaction categories, respectively, outperforming prior work by a large margin.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 70 of its 81 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

70 of 81 references.

References · 70

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  7. 2014Conditional Generative Adversarial Netscited by 2,162
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  9. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  10. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  11. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  12. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  13. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  14. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  15. 2021Variational Diffusion Modelscited by 1,153
  16. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,039
  17. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  18. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  19. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  20. 2020Pre-Trained Image Processing Transformercited by 893
  21. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  22. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
  23. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  24. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 565
  25. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  26. 2023Zero-shot Image-to-Image Translationcited by 505
  27. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  28. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  29. 2023Universal Guidance for Diffusion Modelscited by 437
  30. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  31. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  32. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  33. 2017Toward Multimodal Image-to-Image Translationcited by 296
  34. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  35. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulationscited by 244
  36. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  37. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 204
  38. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 20; 2026: 7.

40 papers.

Sort:

By area:

  1. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  2. 2024Video as the New Language for Real-World Decision Makingcited by 87
  3. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  4. 2024SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videoscited by 24
  5. 2024GenEx: Generating an Explorable Worldcited by 21
  6. 2025EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memorycited by 20
  7. 2025Precise Action-to-Video Generation Through Visual Action Promptscited by 15
  8. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  9. 2024Generative World Explorercited by 12
  10. 2025Mask2IV: Interaction-Centric Video Generation via Mask Trajectoriescited by 10
  11. 2024ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructionscited by 8
  12. 2024VEDIT: Latent Prediction Architecture For Procedural Video Representation Learningcited by 7
28 more
  1. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  2. 2025The Promise of RL for Autoregressive Image Editingcited by 6
  3. 2024HANDI: Hand-Centric Text-and-Image Conditioned Video Generationcited by 5
  4. 2025UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depthscited by 4
  5. 2025Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflectioncited by 4
  6. 2025Ego-centric Predictive Model Conditioned on Hand Trajectoriescited by 4
  7. 2026Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimizationcited by 4
  8. 2024Coherent Zero-Shot Visual Instruction Generationcited by 3
  9. 2024ACDC: Autoregressive Coherent Multimodal Generation using Diffusion Correctioncited by 3
  10. 2025SPOC: Spatially-Progressing Object State Change Segmentation in Videocited by 3
  11. 2026OSCBench: Benchmarking Object State Change in Text-to-Video Generationcited by 3
  12. 2024Anticipating Object State Changes in Long Procedural Videoscited by 2
  13. 2025Latent Beam Diffusion Models for Generating Visual Sequencescited by 2
  14. 2025Enhance Multimodal Consistency and Coherence for Text-Image Plan Generationcited by 2
  15. 2025Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progresscited by 2
  16. 2025Show Me: Unifying Instructional Image and Video Generation with Diffusion Modelscited by 2
  17. 2024VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Promptingcited by 1
  18. 2025Learning Human Skill Generators at Key-Step Levelscited by 1
  19. 2025Stitch-a-Demo: Video Demonstrations from Multistep Descriptionscited by 1
  20. 2025Towards Consistent Long-Term Pose Generationcited by 1
  21. 2026UniMPA: A Unified Memory-Prediction-Action Model via Action-Grounded Transition Modelingcited by 1
  22. 2025VisualChef: Generating Visual Aids in Cooking via Mask Inpaintingcited by 0
  23. 2025Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challengescited by 0
  24. 2025Do-Undo Bench: Reversibility for Action Understanding in Image Generationcited by 0
  25. 2026Ego-InBetween: Generating Object State Transitions in Ego-Centric Videoscited by 0
  26. 2026DenseStep2M: A Scalable, Training-Free Pipeline for Dense Instructional Video Annotationcited by 0
  27. 2026Diagnosing Under-Development of Irreversible Processes in Video Generationcited by 0
  28. 2026LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedurescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SneakPeek: Future-Guided Instructional Streaming Video Generation
  2. 2025GenHSI: Controllable Generation of Human-Scene Interaction Videos
  3. 2025Target-Aware Video Diffusion Models
  4. 2025Image Generation as a Visual Planner for Robotic Manipulation
  5. 2025CoVAR: Co-generation of Video and Action for Robotic Manipulation via Multi-Modal Diffusion
  6. 2025ProcObject-10K: Benchmarking Object-Centric Procedural Understanding in Instructional Videos
  7. 2025MOVi: Training-free Text-conditioned Multi-Object Video Generation
  8. 2025Many-for-Many: Unify the Training of Multiple Video and Image Generation and Manipulation Tasks