# Self-Supervised Learning of Perceptually Optimized Block Motion Estimates for Video Compression

arXiv: https://arxiv.org/abs/2110.01805 · HTML: https://mlsanity.com/paper/2110.01805/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 56 references that are in the map.
- First posted on arXiv 5 Oct 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [ProxIQA: A Proxy Approach to Perceptual Optimization of Learned Image Compression](https://mlsanity.com/paper/1910.08845/) (2019)
- [What are the Receptive, Effective Receptive, and Projective Fields of Neurons in Convolutional Neural Networks?](https://mlsanity.com/paper/1705.07049/) (2017)
- [Reducing Complexity of HEVC: A Deep Learning Approach](https://mlsanity.com/paper/1710.01218/) (2017)
- [Learning to Compress Videos without Computing Motion](https://mlsanity.com/paper/2009.14110/) (2020)
- [Affine Transformation-Based Deep Frame Prediction](https://mlsanity.com/paper/2009.05666/) (2020)
- [Speeding up VP9 Intra Encoder with Hierarchical Deep Learning Based Partition Prediction](https://mlsanity.com/paper/1906.06476/) (2019)

## Built on by (most cited first, 2 of 2)

- [Estimating the Resize Parameter in End-to-end Learned Image Compression](https://mlsanity.com/paper/2204.12022/) (2022)
- [QR-CLIP: Introducing Explicit Open-World Knowledge for Location and Time Reasoning](https://mlsanity.com/paper/2302.00952/) (2023)

## Abstract (arXiv)

Block based motion estimation is integral to inter prediction processes performed in hybrid video codecs. Prevalent block matching based methods that are used to compute block motion vectors (MVs) rely on computationally intensive search procedures. They also suffer from the aperture problem, which can worsen as the block size is reduced. Moreover, the block matching criteria used in typical codecs do not account for the resulting levels of perceptual quality of the motion compensated pictures that are created upon decoding. Towards achieving the elusive goal of perceptually optimized motion estimation, we propose a search-free block motion estimation framework using a multi-stage convolutional neural network, which is able to conduct motion estimation on multiple block sizes simultaneously, using a triplet of frames as input. This composite block translation network (CBT-Net) is trained in a self-supervised manner on a large database that we created from publicly available uncompressed video content. We deploy the multi-scale structural similarity (MS-SSIM) loss function to optimize the perceptual quality of the motion compensated predicted frames. Our experimental results highlight the computational efficiency of our proposed model relative to conventional block matching based motion estimation algorithms, for comparable prediction errors. Further, when used to perform inter prediction in AV1, the MV predictions of the perceptually optimized model result in average Bjontegaard-delta rate (BD-rate) improvements of -1.70% and -1.52% with respect to the MS-SSIM and Video Multi-Method Assessment Fusion (VMAF) quality metrics, respectively as compared to the block matching based motion estimation system employed in the SVT-AV1 encoder.