# Learning Fine-Grained Motion Embedding for Landscape Animation

arXiv: https://arxiv.org/abs/2109.02216 · HTML: https://mlsanity.com/paper/2109.02216/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [High-Resolution Representations for Labeling Pixels and Regions](https://mlsanity.com/paper/1904.04514/) (2019)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting](https://mlsanity.com/paper/1904.07475/) (2019)
- [Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks](https://mlsanity.com/paper/1709.07592/) (2017)
- [Flow-Grounded Spatial-Temporal Video Prediction from Still Images](https://mlsanity.com/paper/1807.09755/) (2018)
- [Video Generation from Single Semantic Label Map](https://mlsanity.com/paper/1903.04480/) (2019)
- [Learning to Forecast and Refine Residual Motion for Image-to-Video Generation](https://mlsanity.com/paper/1807.09951/) (2018)
- [DeepLandscape: Adversarial Modeling of Landscape Video](https://mlsanity.com/paper/2008.09655/) (2020)
- [Animating Landscape: Self-Supervised Learning of Decoupled Motion and Appearance for Single-Image Video Synthesis](https://mlsanity.com/paper/1910.07192/) (2019)
- [End-to-End Time-Lapse Video Synthesis from a Single Outdoor Image](https://mlsanity.com/paper/1904.00680/) (2019)
- [Learning to Take Directions One Step at a Time](https://mlsanity.com/paper/1812.01874/) (2018)
- [Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks](https://mlsanity.com/paper/1801.09042/) (2018)

## Built on by (most cited first, 7 of 7)

- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022)
- [ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation](https://mlsanity.com/paper/2406.18522/) (2024)
- [MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators](https://mlsanity.com/paper/2404.05014/) (2024)
- [Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers](https://mlsanity.com/paper/2111.03481/) (2021)
- [Language-Guided Face Animation by Recurrent StyleGAN-based Generator](https://mlsanity.com/paper/2208.05617/) (2022)
- [A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches](https://mlsanity.com/paper/2401.14718/) (2024)
- [TimeNeRF: Building Generalizable Neural Radiance Fields across Time from Few-Shot Input Views](https://mlsanity.com/paper/2507.13929/) (2025)

## Abstract (arXiv)

In this paper we focus on landscape animation, which aims to generate time-lapse videos from a single landscape image. Motion is crucial for landscape animation as it determines how objects move in videos. Existing methods are able to generate appealing videos by learning motion from real time-lapse videos. However, current methods suffer from inaccurate motion generation, which leads to unrealistic video results. To tackle this problem, we propose a model named FGLA to generate high-quality and realistic videos by learning Fine-Grained motion embedding for Landscape Animation. Our model consists of two parts: (1) a motion encoder which embeds time-lapse motion in a fine-grained way. (2) a motion generator which generates realistic motion to animate input images. To train and evaluate on diverse time-lapse videos, we build the largest high-resolution Time-lapse video dataset with Diverse scenes, namely Time-lapse-D, which includes 16,874 video clips with over 10 million frames. Quantitative and qualitative experimental results demonstrate the superiority of our method. In particular, our method achieves relative improvements by 19% on LIPIS and 5.6% on FVD compared with state-of-the-art methods on our dataset. A user study carried out with 700 human subjects shows that our approach visually outperforms existing methods by a large margin.