# Fast and Accurate: Video Enhancement using Sparse Depth

arXiv: https://arxiv.org/abs/2103.08764 · HTML: https://mlsanity.com/paper/2103.08764/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 15 Mar 2021.

## Builds on (most cited first, 20 of 20)

- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [TDAN: Temporally Deformable Alignment Network for Video Super-Resolution](https://mlsanity.com/paper/1812.02898/) (2018)
- [3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection](https://mlsanity.com/paper/2004.12636/) (2020)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation](https://mlsanity.com/paper/1611.05250/) (2016)
- [Recurrent Back-Projection Network for Video Super-Resolution](https://mlsanity.com/paper/1903.10128/) (2019)
- [DVDnet: A Fast Network for Deep Video Denoising](https://mlsanity.com/paper/1906.11890/) (2019)
- [Sensor Fusion for Joint 3D Object Detection and Semantic Segmentation](https://mlsanity.com/paper/1904.11466/) (2019)
- [ViDeNN: Deep Blind Video Denoising](https://mlsanity.com/paper/1904.10898/) (2019)
- [RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/1906.00208/) (2019)
- [Noise-Aware Unsupervised Deep Lidar-Stereo Fusion](https://mlsanity.com/paper/1904.03868/) (2019)
- [3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization](https://mlsanity.com/paper/1904.02917/) (2019)
- [Gyroscope-Aided Motion Deblurring with Deep Networks](https://mlsanity.com/paper/1810.00986/) (2018)
- [Real-Time Spatio-Temporal LiDAR Point Cloud Compression](https://mlsanity.com/paper/2008.06972/) (2020)
- [Learning-based Image Enhancement for Visual Odometry in Challenging HDR Environments](https://mlsanity.com/paper/1707.01274/) (2017)

## Built on by (most cited first, 2 of 2)

- [DAVIDE: Depth-Aware Video Deblurring](https://mlsanity.com/paper/2409.01274/) (2024)
- [Deltoris: Enabling Real-time VLA Inference in Embodied AI via Bit-level Sparsity and Speculative Inference](https://mlsanity.com/paper/2608.04428/) (2026)

## Abstract (arXiv)

This paper presents a general framework to build fast and accurate algorithms for video enhancement tasks such as super-resolution, deblurring, and denoising. Essential to our framework is the realization that the accuracy, rather than the density, of pixel flows is what is required for high-quality video enhancement. Most of prior works take the opposite approach: they estimate dense (per-pixel)-but generally less robust-flows, mostly using computationally costly algorithms. Instead, we propose a lightweight flow estimation algorithm; it fuses the sparse point cloud data and (even sparser and less reliable) IMU data available in modern autonomous agents to estimate the flow information. Building on top of the flow estimation, we demonstrate a general framework that integrates the flows in a plug-and-play fashion with different task-specific layers. Algorithms built in our framework achieve 1.78x - 187.41x speedup while providing a 0.42 dB - 6.70 dB quality improvement over competing methods.