# Full Reference Video Quality Assessment for Machine Learning-Based Video Codecs

arXiv: https://arxiv.org/abs/2309.00769 · HTML: https://mlsanity.com/paper/2309.00769/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 54 references that are in the map.
- First posted on arXiv 2 Sep 2023.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Quality Assessment of In-the-Wild Videos](https://mlsanity.com/paper/1908.00375/) (2019)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [A Deep Learning based No-reference Quality Assessment Model for UGC Videos](https://mlsanity.com/paper/2204.14047/) (2022)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [Better Aggregation in Test-Time Augmentation](https://mlsanity.com/paper/2011.11156/) (2020)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Blindly Assess Quality of In-the-Wild Videos via Quality-aware Pre-training and Motion Perception](https://mlsanity.com/paper/2108.08505/) (2021)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [A Technical Overview of AV1](https://mlsanity.com/paper/2008.06091/) (2020)
- [C3DVQA: Full-Reference Video Quality Assessment with 3D Convolutional Neural Network](https://mlsanity.com/paper/1910.13646/) (2019)
- [Recover Subjective Quality Scores from Noisy Measurements](https://mlsanity.com/paper/1611.01715/) (2016)
- [A crowdsourcing approach to video quality assessment](https://mlsanity.com/paper/2204.06784/) (2022)
- [Transformation of Mean Opinion Scores to Avoid Misleading of Ranked based Statistical Techniques](https://mlsanity.com/paper/2004.11490/) (2020)
- [VMAF And Variants: Towards A Unified VQA](https://mlsanity.com/paper/2103.07770/) (2021)

## Built on by (most cited first, 2 of 2)

- [A multidimensional measurement of photorealistic avatar quality of experience](https://mlsanity.com/paper/2411.09066/) (2024)
- [FDIM: A Feature-distance-based Generic Video Quality Metric for Versatile Codecs](https://mlsanity.com/paper/2604.24123/) (2026)

## Abstract (arXiv)

Machine learning-based video codecs have made significant progress in the past few years. A critical area in the development of ML-based video codecs is an accurate evaluation metric that does not require an expensive and slow subjective test. We show that existing evaluation metrics that were designed and trained on DSP-based video codecs are not highly correlated to subjective opinion when used with ML video codecs due to the video artifacts being quite different between ML and video codecs. We provide a new dataset of ML video codec videos that have been accurately labeled for quality. We also propose a new full reference video quality assessment (FRVQA) model that achieves a Pearson Correlation Coefficient (PCC) of 0.99 and a Spearman's Rank Correlation Coefficient (SRCC) of 0.99 at the model level. We make the dataset and FRVQA model open source to help accelerate research in ML video codecs, and so that others can further improve the FRVQA model.