# Controllable Animation of Fluid Elements in Still Images

arXiv: https://arxiv.org/abs/2112.03051 · HTML: https://mlsanity.com/paper/2112.03051/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 26 of 26)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [Stochastic Latent Residual Video Prediction](https://mlsanity.com/paper/2002.09219/) (2020)
- [Improved Conditional VRNNs for Video Prediction](https://mlsanity.com/paper/1904.12165/) (2019)
- [Unsupervised Learning of Object Structure and Dynamics from Videos](https://mlsanity.com/paper/1906.07889/) (2019)
- [Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks](https://mlsanity.com/paper/1709.07592/) (2017)
- [High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks](https://mlsanity.com/paper/1911.01655/) (2019)
- [Future Video Synthesis with Object Motion Prediction](https://mlsanity.com/paper/2004.00542/) (2020)
- [Flow-Grounded Spatial-Temporal Video Prediction from Still Images](https://mlsanity.com/paper/1807.09755/) (2018)
- [Animating Pictures with Eulerian Motion Fields](https://mlsanity.com/paper/2011.15128/) (2020)
- [Video Generation from Single Semantic Label Map](https://mlsanity.com/paper/1903.04480/) (2019)
- [SDCNet: Video Prediction Using Spatially-Displaced Convolution](https://mlsanity.com/paper/1811.00684/) (2018)
- [Stochastic Image-to-Video Synthesis using cINNs](https://mlsanity.com/paper/2105.04551/) (2021)
- [iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis](https://mlsanity.com/paper/2107.02790/) (2021)
- [Understanding Object Dynamics for Interactive Image-to-Video Synthesis](https://mlsanity.com/paper/2106.11303/) (2021)
- [DeepLandscape: Adversarial Modeling of Landscape Video](https://mlsanity.com/paper/2008.09655/) (2020)
- [Animating Landscape: Self-Supervised Learning of Decoupled Motion and Appearance for Single-Image Video Synthesis](https://mlsanity.com/paper/1910.07192/) (2019)

## Built on by (most cited first, 30 of 45)

- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023)
- [PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation](https://mlsanity.com/paper/2409.18964/) (2024)
- [Generative Image Dynamics](https://mlsanity.com/paper/2309.07906/) (2023)
- [MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model](https://mlsanity.com/paper/2405.20222/) (2024)
- [AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidance](https://mlsanity.com/paper/2311.12886/) (2023)
- [PhysGen3D: Crafting a Miniature Interactive World from a Single Image](https://mlsanity.com/paper/2503.20746/) (2025)
- [LivePhoto: Real Image Animation with Text-guided Motion Control](https://mlsanity.com/paper/2312.02928/) (2023)
- [FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis](https://mlsanity.com/paper/2502.08244/) (2025)
- [TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models](https://mlsanity.com/paper/2403.17005/) (2024)
- [StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration](https://mlsanity.com/paper/2411.04925/) (2024)
- [TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models](https://mlsanity.com/paper/2404.16306/) (2024)
- [3D Cinemagraphy from a Single Image](https://mlsanity.com/paper/2303.05724/) (2023)
- [Text-Guided Synthesis of Eulerian Cinemagraphs](https://mlsanity.com/paper/2307.03190/) (2023)
- [Explorative Inbetweening of Time and Space](https://mlsanity.com/paper/2403.14611/) (2024)
- [VividDream: Generating 3D Scene with Ambient Dynamics](https://mlsanity.com/paper/2405.20334/) (2024)
- [iNVS: Repurposing Diffusion Inpainters for Novel View Synthesis](https://mlsanity.com/paper/2310.16167/) (2023)
- [Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Image](https://mlsanity.com/paper/2308.10257/) (2023)
- [FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation](https://mlsanity.com/paper/2505.10075/) (2025)
- [MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation](https://mlsanity.com/paper/2412.05848/) (2024)
- [Simulating Fluids in Real-World Still Images](https://mlsanity.com/paper/2204.11335/) (2022)
- [Blowing in the Wind: CycleNet for Human Cinemagraphs from Still Images](https://mlsanity.com/paper/2303.08639/) (2023)
- [Dance Any Beat: Blending Beats with Visuals in Dance Video Generation](https://mlsanity.com/paper/2405.09266/) (2024)
- [Automatic Animation of Hair Blowing in Still Portrait Photos](https://mlsanity.com/paper/2309.14207/) (2023)
- [LoopGaussian: Creating 3D Cinemagraph with Multi-view Images via Eulerian Motion Field](https://mlsanity.com/paper/2404.08966/) (2024)
- [A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches](https://mlsanity.com/paper/2401.14718/) (2024)
- [Masked Generative Priors Improve World Models Sequence Modelling Capabilities](https://mlsanity.com/paper/2410.07836/) (2024)
- [3D Object Manipulation in a Single Image using Generative Models](https://mlsanity.com/paper/2501.12935/) (2025)
- [3D Video Loops from Asynchronous Input](https://mlsanity.com/paper/2303.05312/) (2023)

## Abstract (arXiv)

We propose a method to interactively control the animation of fluid elements in still images to generate cinemagraphs. Specifically, we focus on the animation of fluid elements like water, smoke, fire, which have the properties of repeating textures and continuous fluid motion. Taking inspiration from prior works, we represent the motion of such fluid elements in the image in the form of a constant 2D optical flow map. To this end, we allow the user to provide any number of arrow directions and their associated speeds along with a mask of the regions the user wants to animate. The user-provided input arrow directions, their corresponding speed values, and the mask are then converted into a dense flow map representing a constant optical flow map (FD). We observe that FD, obtained using simple exponential operations can closely approximate the plausible motion of elements in the image. We further refine computed dense optical flow map FD using a generative-adversarial network (GAN) to obtain a more realistic flow map. We devise a novel UNet based architecture to autoregressively generate future frames using the refined optical flow map by forward-warping the input image features at different resolutions. We conduct extensive experiments on a publicly available dataset and show that our method is superior to the baselines in terms of qualitative and quantitative metrics. In addition, we show the qualitative animations of the objects in directions that did not exist in the training set and provide a way to synthesize videos that otherwise would not exist in the real world.