Show Me: Unifying Instructional Image and Video Generation with Diffusion Models
On the map: Diffusion models → Video generation
Abstract (arXiv)
Generating visual instructions in a given context is essential for developing interactive world simulators. While prior works address this problem through either text-guided image manipulation or video prediction, these tasks are typically treated in isolation. This separation reveals a fundamental issue: image manipulation methods overlook how actions unfold over time, while video prediction models often ignore the intended outcomes. To this end, we propose ShowMe, a unified framework that enables both tasks by selectively activating the spatial and temporal components of video diffusion models. In addition, we introduce structure and motion consistency rewards to improve structural fidelity and temporal coherence. Notably, this unification brings dual benefits: the spatial knowledge gained through video pretraining enhances contextual consistency and realism in non-rigid image edits, while the instruction-guided manipulation stage equips the model with stronger goal-oriented reasoning for video prediction. Experiments on diverse benchmarks demonstrate that our method outperforms expert models in both instructional image and video generation, highlighting the strength of video diffusion models as a unified action-object state transformer.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 67 of its 78 references that are in the map. First posted on arXiv 21 Nov 2025.
Builds on
67 of 78 references.
Closest ancestors · 22
Same direction, newest first
- 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise14 Jan
- 2024AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction10 Jun
- 2024Video Interpolation with Diffusion Models1 Apr
- 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation28 Nov
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
- 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
- 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
- 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors18 Oct
- 2023CCEdit: Creative and Controllable Video Editing via Diffusion Models28 Sep
- 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing18 Aug
- 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet26 Jul
- 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
10 more
- 2023Towards Consistent Video Editing with Text-to-Image Diffusion Models27 May
- 2023VDT: General-purpose Video Diffusion Transformers via Mask Modeling22 May
- 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
- 2023Pix2Video: Video Editing using Image Diffusion22 Mar
- 2023LDMVFI: Video Frame Interpolation with Latent Diffusion Models16 Mar
- 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
- 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
- 2022Diffusion Models for Video Prediction and Infilling15 Jun
- 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
- 2022Video Diffusion Models7 Apr
Other references · 45
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2023GPT-4 Technical Reportcited by 18,132
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2020Denoising Diffusion Implicit Modelscited by 9,655
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 2023Visual Instruction Tuningcited by 7,792
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
33 more
- 2022Classifier-Free Diffusion Guidancecited by 5,717
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
- 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
- 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
- 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
- 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
- 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
- 2015Holistically-Nested Edge Detectioncited by 590
- 2020Rescaling Egocentric Visioncited by 529
- 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
- 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
- 2023Learning Interactive Real-World Simulatorscited by 440
- 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
- 2023Universal Guidance for Diffusion Modelscited by 426
- 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
- 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
- 2022Egocentric Video-Language Pretrainingcited by 231
- 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
- 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
- 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
- 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
- 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
- 2023AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidancecited by 66
- 2023Seer: Language Instructed Video Prediction with Latent Diffusion Modelscited by 51
- 2023GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videoscited by 39
- 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
- 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
- 2023Generating Illustrated Instructionscited by 7
- 2024ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructionscited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025SneakPeek: Future-Guided Instructional Streaming Video Generation
- 2026InstructionCrafter: Generating Consistent and High-Fidelity Visual Instructions
- 2026Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation
- 2025Target-Aware Video Diffusion Models
- 2024Animate Your Motion: Turning Still Images into Dynamic Videos
- 2025From Image to Video: An Empirical Study of Diffusion Representations
- 2025Image Generation as a Visual Planner for Robotic Manipulation
- 2026CausalMotion: Structured Physical Reasoning as Keyframe and Trajectory Guidance for Training-Free Video Generation