# Discovering Limitations of Image Quality Assessments with Noised Deep Learning Image Sets

arXiv: https://arxiv.org/abs/2210.10249 · HTML: https://mlsanity.com/paper/2210.10249/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 16 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 2 of 2)

- [Understanding How Image Quality Affects Deep Neural Networks](https://mlsanity.com/paper/1604.04004/) (2016)
- [Benchmarking Robustness of Deep Learning Classifiers Using Two-Factor Perturbation](https://mlsanity.com/paper/2103.03102/) (2021)

## Built on by (most cited first, 2 of 2)

- [Visual Question Answering (VQA) on Images with Superimposed Text](https://mlsanity.com/paper/2307.02489/) (2023)
- [ASI: Accuracy-Stability Index for Evaluating Deep Learning Models](https://mlsanity.com/paper/2311.15332/) (2023)

## Abstract (arXiv)

Image quality is important, and can affect overall performance in image processing and computer vision as well as for numerous other reasons. Image quality assessment (IQA) is consequently a vital task in different applications from aerial photography interpretation to object detection to medical image analysis. In previous research, the BRISQUE algorithm and the PSNR algorithm were evaluated with high resolution (atleast 512x384 pixels), but relatively small image sets (no more than 4,744 images). However, scientists have not evaluated IQA algorithms on low resolution (no more than 32x32 pixels), multi-perturbation, big image sets (for example, tleast 60,000 different images not counting their perturbations). This study explores these two IQA algorithms through experimental investigation. We first chose two deep learning image sets, CIFAR-10 and MNIST. Then, we added 68 perturbations that add noise to the images in specific sequences and noise intensities. In addition, we tracked the performance outputs of the two IQA algorithms with singly and multiply noised images. After quantitatively analyzing experimental results, we report the limitations of the two IQAs with these noised CIFAR-10 and MNIST image sets. We also explain three potential root causes for performance degradation. These findings point out weaknesses of the two IQA algorithms. The research results provide guidance to scientists and engineers developing accurate, robust IQA algorithms. All source codes, related image sets, and figures are shared on the website (https://github.com/caperock/imagequality) to support future scientific and industrial projects.