# Blocks World Revisited: The Effect of Self-Occlusion on Classification by Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2102.12911 · HTML: https://mlsanity.com/paper/2102.12911/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 64 references that are in the map.
- First posted on arXiv 25 Feb 2021.

## Builds on (most cited first, 18 of 18)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation](https://mlsanity.com/paper/1808.01838/) (2018)
- [Compositional Convolutional Neural Networks: A Robust and Interpretable Model for Object Recognition under Occlusion](https://mlsanity.com/paper/2006.15538/) (2020)
- [Deep Supervision with Intermediate Concepts](https://mlsanity.com/paper/1801.03399/) (2018)
- [Instance- and Category-level 6D Object Pose Estimation](https://mlsanity.com/paper/1903.04229/) (2019)
- [SIDOD: A Synthetic Image Dataset for 3D Object Pose Recognition with Distractors](https://mlsanity.com/paper/2008.05955/) (2020)
- [Random Polyhedral Scenes: An Image Generator for Active Vision System Experiments](https://mlsanity.com/paper/1803.10100/) (2018)

## Built on by (most cited first, 4 of 4)

- [The Psychophysics of Human Three-Dimensional Active Visuospatial Problem-Solving](https://mlsanity.com/paper/2306.11756/) (2023)
- [Active Observer Visual Problem-Solving Methods are Dynamically Hypothesized, Deployed and Tested](https://mlsanity.com/paper/2108.08145/) (2021)
- [Next-Best-View Estimation based on Deep Reinforcement Learning for Active Object Classification](https://mlsanity.com/paper/2110.06766/) (2021)
- [Boosting Reinforcement Learning in 3D Visuospatial Tasks Through Human-Informed Curriculum Design](https://mlsanity.com/paper/2511.17595/) (2025)

## Abstract (arXiv)

Despite the recent successes in computer vision, there remain new avenues to explore. In this work, we propose a new dataset to investigate the effect of self-occlusion on deep neural networks. With TEOS (The Effect of Self-Occlusion), we propose a 3D blocks world dataset that focuses on the geometric shape of 3D objects and their omnipresent challenge of self-occlusion. We designed TEOS to investigate the role of self-occlusion in the context of object classification. Even though remarkable progress has been seen in object classification, self-occlusion is a challenge. In the real-world, self-occlusion of 3D objects still presents significant challenges for deep learning approaches. However, humans deal with this by deploying complex strategies, for instance, by changing the viewpoint or manipulating the scene to gather necessary information. With TEOS, we present a dataset of two difficulty levels (L1 and L2 ), containing 36 and 12 objects, respectively. We provide 738 uniformly sampled views of each object, their mask, object and camera position, orientation, amount of self-occlusion, as well as the CAD model of each object. We present baseline evaluations with five well-known classification deep neural networks and show that TEOS poses a significant challenge for all of them. The dataset, as well as the pre-trained models, are made publicly available for the scientific community under https://nvision2.data.eecs.yorku.ca/TEOS.