# Self-Supervised Motion Magnification by Backpropagating Through Optical Flow

arXiv: https://arxiv.org/abs/2311.17056 · HTML: https://mlsanity.com/paper/2311.17056/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 69 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 30 of 37)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [Test-Time Training with Masked Autoencoders](https://mlsanity.com/paper/2209.07522/) (2022)
- [TAO: A Large-Scale Benchmark for Tracking Any Object](https://mlsanity.com/paper/2005.10356/) (2020)
- [Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation](https://mlsanity.com/paper/2104.04691/) (2021)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)
- [FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation](https://mlsanity.com/paper/2012.08512/) (2020)
- [DDFlow: Learning Optical Flow with Unlabeled Data Distillation](https://mlsanity.com/paper/1902.09145/) (2019)

## Built on by (most cited first, 10 of 10)

- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)
- [Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators](https://mlsanity.com/paper/2401.18085/) (2024)
- [Learning-based Axial Video Motion Magnification](https://mlsanity.com/paper/2312.09551/) (2023)
- [Independently Keypoint Learning for Small Object Semantic Correspondence](https://mlsanity.com/paper/2404.02678/) (2024)
- [Deep-learning Optical Flow Outperforms PIV in Obtaining Velocity Fields from Active Nematics](https://mlsanity.com/paper/2404.15497/) (2024)
- [SpikeMM: Flexi-Magnification of High-Speed Micro-Motions](https://mlsanity.com/paper/2406.00383/) (2024)
- [AMMSM: Adaptive Motion Magnification and Sparse Mamba for Micro-Expression Recognition](https://mlsanity.com/paper/2503.24057/) (2025)
- [EndoControlMag: Robust Endoscopic Vascular Motion Magnification with Periodic Reference Resetting and Hierarchical Tissue-aware Dual-Mask Control](https://mlsanity.com/paper/2507.15292/) (2025)
- [GeoDiffMM: Geometry-Guided Conditional Diffusion for Motion Magnification](https://mlsanity.com/paper/2512.08325/) (2025)
- [GeoMag: Geometric-Aware Video Motion Magnification via State Space Model](https://mlsanity.com/paper/2605.29762/) (2026)

## Abstract (arXiv)

This paper presents a simple, self-supervised method for magnifying subtle motions in video: given an input video and a magnification factor, we manipulate the video such that its new optical flow is scaled by the desired amount. To train our model, we propose a loss function that estimates the optical flow of the generated video and penalizes how far if deviates from the given magnification factor. Thus, training involves differentiating through a pretrained optical flow network. Since our model is self-supervised, we can further improve its performance through test-time adaptation, by finetuning it on the input video. It can also be easily extended to magnify the motions of only user-selected objects. Our approach avoids the need for synthetic magnification datasets that have been used to train prior learning-based approaches. Instead, it leverages the existing capabilities of off-the-shelf motion estimators. We demonstrate the effectiveness of our method through evaluations of both visual quality and quantitative metrics on a range of real-world and synthetic videos, and we show our method works for both supervised and unsupervised optical flow methods.