# Leveraging Motion Estimation for Efficient Bayer-Domain Video Convolutional Networks

arXiv: https://arxiv.org/abs/2501.15119 · HTML: https://mlsanity.com/paper/2501.15119/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 25 Jan 2025.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1904.02216/) (2019)
- [Invertible Image Signal Processing](https://mlsanity.com/paper/2103.15061/) (2021)
- [Accel: A Corrective Fusion Network for Efficient Semantic Segmentation on Video](https://mlsanity.com/paper/1807.06667/) (2018)
- [Temporally Distributed Networks for Fast Video Semantic Segmentation](https://mlsanity.com/paper/2004.01800/) (2020)
- [Low-Latency Video Semantic Segmentation](https://mlsanity.com/paper/1804.00389/) (2018)
- [FIRST: A Framework for Optimizing Information Quality in Mobile Crowdsensing Systems](https://mlsanity.com/paper/1804.11147/) (2018)
- [CameraNet: A Two-Stage Framework for Effective Camera ISP Learning](https://mlsanity.com/paper/1908.01481/) (2019)
- [HighEr-Resolution Network for Image Demosaicing and Enhancing](https://mlsanity.com/paper/1911.08098/) (2019)
- [BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies](https://mlsanity.com/paper/2108.09376/) (2021)
- [Deep Camera: A Fully Convolutional Neural Network for Image Signal Processing](https://mlsanity.com/paper/1908.09191/) (2019)
- [Efficient Semantic Segmentation by Altering Resolutions for Compressed Videos](https://mlsanity.com/paper/2303.07224/) (2023)
- [Raw Bayer Pattern Image Synthesis for Computer Vision-oriented Image Signal Processing Pipeline Design](https://mlsanity.com/paper/2110.12823/) (2021)

## Abstract (arXiv)

Existing computer vision pipelines suffer from two efficiency bottlenecks: image signal processors (ISPs) first convert Bayer pixel data to RGB pixel by pixel, and video models repeatedly process temporally redundant frames, leading to high power consumption and latency. In this paper, we propose a novel framework that eliminates the ISP and leverages motion estimation to accelerate video vision tasks directly in the Bayer domain. We introduce Motion Estimation-based Video Convolution (MEVC), which integrates sliding-window motion estimation into each convolutional layer, enabling prediction and residual-based refinement that reduces redundant computations across frames. This design bridges the structural gap between block-based motion estimation and spatial convolution, enabling accurate, low-cost processing. Our end-to-end pipeline supports raw Bayer input and achieves over 70% reduction in FLOPs with minimal accuracy degradation across video semantic segmentation, depth estimation, and object detection benchmarks, using both synthetic Bayer-converted and real Bayer video datasets.