# A Structured Analysis of the Video Degradation Effects on the Performance of a Machine Learning-enabled Pedestrian Detector

arXiv: https://arxiv.org/abs/2106.15889 · HTML: https://mlsanity.com/paper/2106.15889/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 12 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 7 of 7)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Effects of Degradations on Deep Neural Network Architectures](https://mlsanity.com/paper/1807.10108/) (2018)
- [Google's Cloud Vision API Is Not Robust To Noise](https://mlsanity.com/paper/1704.05051/) (2017)
- [Towards Robust Neural Networks with Lipschitz Continuity](https://mlsanity.com/paper/1811.09008/) (2018)
- [Improving Image Classification Robustness through Selective CNN-Filters Fine-Tuning](https://mlsanity.com/paper/1904.03949/) (2019)

## Abstract (arXiv)

ML-enabled software systems have been incorporated in many public demonstrations for automated driving (AD) systems. Such solutions have also been considered as a crucial approach to aim at SAE Level 5 systems, where the passengers in such vehicles do not have to interact with the system at all anymore. Already in 2016, Nvidia demonstrated a complete end-to-end approach for training the complete software stack covering perception, planning and decision making, and the actual vehicle control. While such approaches show the great potential of such ML-enabled systems, there have also been demonstrations where already changes to single pixels in a video frame can potentially lead to completely different decisions with dangerous consequences. In this paper, a structured analysis has been conducted to explore video degradation effects on the performance of an ML-enabled pedestrian detector. Firstly, a baseline of applying YOLO to 1,026 frames with pedestrian annotations in the KITTI Vision Benchmark Suite has been established. Next, video degradation candidates for each of these frames were generated using the leading video codecs libx264, libx265, Nvidia HEVC, and AV1: 52 frames for the various compression presets for color and gray-scale frames resulting in 104 degradation candidates per original KITTI frame and 426,816 images in total. YOLO was applied to each image to compute the intersection-over-union (IoU) metric to compare the performance with the original baseline. While aggressively lossy compression settings result in significant performance drops as expected, it was also observed that some configurations actually result in slightly better IoU results compared to the baseline. The findings show that carefully chosen lossy video configurations preserve a decent performance of particular ML-enabled systems while allowing for substantial savings when storing or transmitting data.