# Are Deep Learning Models Robust to Partial Object Occlusion in Visual Recognition Tasks?

arXiv: https://arxiv.org/abs/2409.10775 · HTML: https://mlsanity.com/paper/2409.10775/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Occluded Video Instance Segmentation: A Benchmark](https://mlsanity.com/paper/2102.01558/) (2021)
- [Robust Classification with Convolutional Prototype Learning](https://mlsanity.com/paper/1805.03438/) (2018)
- [Compositional Convolutional Neural Networks: A Deep Architecture with Innate Robustness to Partial Occlusion](https://mlsanity.com/paper/2003.04490/) (2020)
- [Combining Compositional Models and Deep Networks For Robust Object Classification under Occlusion](https://mlsanity.com/paper/1905.11826/) (2019)
- [Recurrent computations for visual pattern completion](https://mlsanity.com/paper/1706.02240/) (2017)
- [Deep Feature Augmentation for Occluded Image Classification](https://mlsanity.com/paper/2011.00768/) (2020)
- [Robustness of Object Recognition under Extreme Occlusion in Humans and Computational Models](https://mlsanity.com/paper/1905.04598/) (2019)
- [Detecting Semantic Parts on Partially Occluded Objects](https://mlsanity.com/paper/1707.07819/) (2017)
- [TDAPNet: Prototype Network with Recurrent Top-Down Attention for Robust Object Classification under Partial Occlusion](https://mlsanity.com/paper/1909.03879/) (2019)

## Built on by (most cited first, 5 of 5)

- [CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Counting](https://mlsanity.com/paper/2504.15485/) (2025)
- [SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation](https://mlsanity.com/paper/2602.23359/) (2026)
- [FLYNN: Robust Neural Network for Robot Navigation using Fly Brain Topology](https://mlsanity.com/paper/2607.00025/) (2026)
- [Improving Classification of Occluded Objects through Scene Context](https://mlsanity.com/paper/2510.26681/) (2025)
- [OASIC: Occlusion-Agnostic and Severity-Informed Classification](https://mlsanity.com/paper/2604.04012/) (2026)

## Abstract (arXiv)

Image classification models, including convolutional neural networks (CNNs), perform well on a variety of classification tasks but struggle under conditions of partial occlusion, i.e., conditions in which objects are partially covered from the view of a camera. Methods to improve performance under occlusion, including data augmentation, part-based clustering, and more inherently robust architectures, including Vision Transformer (ViT) models, have, to some extent, been evaluated on their ability to classify objects under partial occlusion. However, evaluations of these methods have largely relied on images containing artificial occlusion, which are typically computer-generated and therefore inexpensive to label. Additionally, methods are rarely compared against each other, and many methods are compared against early, now outdated, deep learning models. We contribute the Image Recognition Under Occlusion (IRUO) dataset, based on the recently developed Occluded Video Instance Segmentation (OVIS) dataset (arXiv:2102.01558). IRUO utilizes real-world and artificially occluded images to test and benchmark leading methods' robustness to partial occlusion in visual recognition tasks. In addition, we contribute the design and results of a human study using images from IRUO that evaluates human classification performance at multiple levels and types of occlusion. We find that modern CNN-based models show improved recognition accuracy on occluded images compared to earlier CNN-based models, and ViT-based models are more accurate than CNN-based models on occluded images, performing only modestly worse than human accuracy. We also find that certain types of occlusion, including diffuse occlusion, where relevant objects are seen through "holes" in occluders such as fences and leaves, can greatly reduce the accuracy of deep recognition models as compared to humans, especially those with CNN backbones.