# GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos

arXiv: https://arxiv.org/abs/2312.07322 · HTML: https://mlsanity.com/paper/2312.07322/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 70 of its 81 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Builds on (most cited first, 30 of 70)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)

## Built on by (most cited first, 30 of 40)

- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [Video as the New Language for Real-World Decision Making](https://mlsanity.com/paper/2402.17139/) (2024)
- [Learning Action and Reasoning-Centric Image Editing from Videos and Simulations](https://mlsanity.com/paper/2407.03471/) (2024)
- [SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.01599/) (2024)
- [GenEx: Generating an Explorable World](https://mlsanity.com/paper/2412.09624/) (2024)
- [EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memory](https://mlsanity.com/paper/2510.01183/) (2025)
- [Precise Action-to-Video Generation Through Visual Action Prompts](https://mlsanity.com/paper/2508.13104/) (2025)
- [A Step Toward World Models: A Survey on Robotic Manipulation](https://mlsanity.com/paper/2511.02097/) (2025)
- [Generative World Explorer](https://mlsanity.com/paper/2411.11844/) (2024)
- [Mask2IV: Interaction-Centric Video Generation via Mask Trajectories](https://mlsanity.com/paper/2510.03135/) (2025)
- [ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions](https://mlsanity.com/paper/2412.01987/) (2024)
- [VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning](https://mlsanity.com/paper/2410.03478/) (2024)
- [What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning](https://mlsanity.com/paper/2503.21055/) (2025)
- [The Promise of RL for Autoregressive Image Editing](https://mlsanity.com/paper/2508.01119/) (2025)
- [HANDI: Hand-Centric Text-and-Image Conditioned Video Generation](https://mlsanity.com/paper/2412.04189/) (2024)
- [UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths](https://mlsanity.com/paper/2502.06474/) (2025)
- [Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflection](https://mlsanity.com/paper/2503.13500/) (2025)
- [Ego-centric Predictive Model Conditioned on Hand Trajectories](https://mlsanity.com/paper/2508.19852/) (2025)
- [Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization](https://mlsanity.com/paper/2603.09538/) (2026)
- [Coherent Zero-Shot Visual Instruction Generation](https://mlsanity.com/paper/2406.04337/) (2024)
- [ACDC: Autoregressive Coherent Multimodal Generation using Diffusion Correction](https://mlsanity.com/paper/2410.04721/) (2024)
- [SPOC: Spatially-Progressing Object State Change Segmentation in Video](https://mlsanity.com/paper/2503.11953/) (2025)
- [OSCBench: Benchmarking Object State Change in Text-to-Video Generation](https://mlsanity.com/paper/2603.11698/) (2026)
- [Anticipating Object State Changes in Long Procedural Videos](https://mlsanity.com/paper/2405.12789/) (2024)
- [Latent Beam Diffusion Models for Generating Visual Sequences](https://mlsanity.com/paper/2503.20429/) (2025)
- [Enhance Multimodal Consistency and Coherence for Text-Image Plan Generation](https://mlsanity.com/paper/2506.11380/) (2025)
- [Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress](https://mlsanity.com/paper/2509.24129/) (2025)
- [Show Me: Unifying Instructional Image and Video Generation with Diffusion Models](https://mlsanity.com/paper/2511.17839/) (2025)
- [VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting](https://mlsanity.com/paper/2412.11621/) (2024)
- [Learning Human Skill Generators at Key-Step Levels](https://mlsanity.com/paper/2502.08234/) (2025)

## Abstract (arXiv)

We address the task of generating temporally consistent and physically plausible images of actions and object state transformations. Given an input image and a text prompt describing the targeted transformation, our generated images preserve the environment and transform objects in the initial image. Our contributions are threefold. First, we leverage a large body of instructional videos and automatically mine a dataset of triplets of consecutive frames corresponding to initial object states, actions, and resulting object transformations. Second, equipped with this data, we develop and train a conditioned diffusion model dubbed GenHowTo. Third, we evaluate GenHowTo on a variety of objects and actions and show superior performance compared to existing methods. In particular, we introduce a quantitative evaluation where GenHowTo achieves 88% and 74% on seen and unseen interaction categories, respectively, outperforming prior work by a large margin.