# Deep Nets: What have they ever done for Vision?

arXiv: https://arxiv.org/abs/1805.04025 · HTML: https://mlsanity.com/paper/1805.04025/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2018.

## Built on by (most cited first, 23 of 23)

- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [Label-Free Liver Tumor Segmentation](https://mlsanity.com/paper/2303.14869/) (2023)
- [AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weeks](https://mlsanity.com/paper/2305.09666/) (2023)
- [Learning Disentangled Representations in the Imaging Domain](https://mlsanity.com/paper/2108.12043/) (2021)
- [Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustness](https://mlsanity.com/paper/2203.07653/) (2022)
- [Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions](https://mlsanity.com/paper/2309.07510/) (2023)
- [Classification robustness to common optical aberrations](https://mlsanity.com/paper/2308.15499/) (2023)
- [Simulated Adversarial Testing of Face Recognition Models](https://mlsanity.com/paper/2106.04569/) (2021)
- [Trapped in texture bias? A large scale comparison of deep instance segmentation](https://mlsanity.com/paper/2401.09109/) (2024)
- [Learning Inductive Attention Guidance for Partially Supervised Pancreatic Ductal Adenocarcinoma Prediction](https://mlsanity.com/paper/2105.14773/) (2021)
- [Data efficient deep learning for medical image analysis: A survey](https://mlsanity.com/paper/2310.06557/) (2023)
- [DELAUNAY: a dataset of abstract art for psychophysical and machine learning research](https://mlsanity.com/paper/2201.12123/) (2022)
- [What Is Considered Complete for Visual Recognition?](https://mlsanity.com/paper/2105.13978/) (2021)
- [Structured Generative Models for Scene Understanding](https://mlsanity.com/paper/2302.03531/) (2023)
- [Projection: A Mechanism for Human-like Reasoning in Artificial Intelligence](https://mlsanity.com/paper/2103.13512/) (2021)
- [Not Just Object, But State: Compositional Incremental Learning without Forgetting](https://mlsanity.com/paper/2411.01739/) (2024)
- [A Comprehensive Assessment Benchmark for Rigorously Evaluating Deep Learning Image Classifiers](https://mlsanity.com/paper/2308.04137/) (2023)
- [Making CNNs Interpretable by Building Dynamic Sequential Decision Forests with Top-down Hierarchy Learning](https://mlsanity.com/paper/2106.02824/) (2021)
- [Towards the One Learning Algorithm Hypothesis: A System-theoretic Approach](https://mlsanity.com/paper/2112.02256/) (2021)
- [Exploring Semantic Attributes from A Foundation Model for Federated Learning of Disjoint Label Spaces](https://mlsanity.com/paper/2208.13465/) (2022)
- [UW-CVGAN: UnderWater Image Enhancement with Capsules Vectors Quantization](https://mlsanity.com/paper/2302.01144/) (2023)
- [Compositional Representation Learning for Brain Tumour Segmentation](https://mlsanity.com/paper/2310.06562/) (2023)
- [Dynamic Stream Network for Combinatorial Explosion Problem in Deformable Medical Image Registration](https://mlsanity.com/paper/2512.19486/) (2025)

## Abstract (arXiv)

This is an opinion paper about the strengths and weaknesses of Deep Nets for vision. They are at the heart of the enormous recent progress in artificial intelligence and are of growing importance in cognitive science and neuroscience. They have had many successes but also have several limitations and there is limited understanding of their inner workings. At present Deep Nets perform very well on specific visual tasks with benchmark datasets but they are much less general purpose, flexible, and adaptive than the human visual system. We argue that Deep Nets in their current form are unlikely to be able to overcome the fundamental problem of computer vision, namely how to deal with the combinatorial explosion, caused by the enormous complexity of natural images, and obtain the rich understanding of visual scenes that the human visual achieves. We argue that this combinatorial explosion takes us into a regime where "big data is not enough" and where we need to rethink our methods for benchmarking performance and evaluating vision algorithms. We stress that, as vision algorithms are increasingly used in real world applications, that performance evaluation is not merely an academic exercise but has important consequences in the real world. It is impractical to review the entire Deep Net literature so we restrict ourselves to a limited range of topics and references which are intended as entry points into the literature. The views expressed in this paper are our own and do not necessarily represent those of anybody else in the computer vision community.