# Video Enhancement with Task-Oriented Flow

arXiv: https://arxiv.org/abs/1711.09078 · HTML: https://mlsanity.com/paper/1711.09078/

## Key facts

- Cited by 619 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2017.

## Built on by (most cited first, 30 of 619)

- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [VRT: A Video Restoration Transformer](https://mlsanity.com/paper/2201.12288/) (2022)
- [How Far is Video Generation from World Model: A Physical Law Perspective](https://mlsanity.com/paper/2411.02385/) (2024)
- [AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation](https://mlsanity.com/paper/2304.09790/) (2023)
- [VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolution](https://mlsanity.com/paper/2205.03409/) (2022)
- [Recurrent Video Restoration Transformer with Guided Deformable Attention](https://mlsanity.com/paper/2206.02146/) (2022)
- [Lossy Image Compression with Conditional Diffusion Models](https://mlsanity.com/paper/2209.06950/) (2022)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [FILM: Frame Interpolation for Large Motion](https://mlsanity.com/paper/2202.04901/) (2022)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [Low-Light Image and Video Enhancement Using Deep Learning: A Survey](https://mlsanity.com/paper/2104.10729/) (2021)
- [Modulated Periodic Activations for Generalizable Local Functional Representations](https://mlsanity.com/paper/2104.03960/) (2021)
- [Investigating Tradeoffs in Real-World Video Super-Resolution](https://mlsanity.com/paper/2111.12704/) (2021)
- [Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution](https://mlsanity.com/paper/2312.06640/) (2023)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [XVFI: eXtreme Video Frame Interpolation](https://mlsanity.com/paper/2103.16206/) (2021)
- [VideoBooth: Diffusion-based Video Generation with Image Prompts](https://mlsanity.com/paper/2312.00777/) (2023)
- [Video Frame Interpolation Transformer](https://mlsanity.com/paper/2111.13817/) (2021)
- [Neural Video Compression with Feature Modulation](https://mlsanity.com/paper/2402.17414/) (2024)
- [Video Super-Resolution Transformer](https://mlsanity.com/paper/2106.06847/) (2021)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [Towards An End-to-End Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2204.02663/) (2022)
- [Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision](https://mlsanity.com/paper/2306.11719/) (2023)
- [Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion](https://mlsanity.com/paper/2203.17191/) (2022)
- [IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation](https://mlsanity.com/paper/2205.14620/) (2022)
- [Asymmetric Bilateral Motion Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2108.06815/) (2021)
- [VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution](https://mlsanity.com/paper/2206.04647/) (2022)

## Abstract (arXiv)

Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.