# Which Backbone to Use: A Resource-efficient Domain Specific Comparison for Computer Vision

arXiv: https://arxiv.org/abs/2406.05612 · HTML: https://mlsanity.com/paper/2406.05612/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 9 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation](https://mlsanity.com/paper/2103.10158/) (2021)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [An open access repository of images on plant health to enable the development of mobile disease diagnostics](https://mlsanity.com/paper/1511.08060/) (2015)
- [How Well Do Sparse Imagenet Models Transfer?](https://mlsanity.com/paper/2111.13445/) (2021)
- [A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis](https://mlsanity.com/paper/2108.05930/) (2021)
- [Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification](https://mlsanity.com/paper/2108.13122/) (2021)
- [View Independent Vehicle Make, Model and Color Recognition Using Convolutional Neural Network](https://mlsanity.com/paper/1702.01721/) (2017)

## Built on by (most cited first, 6 of 6)

- [Comparative Analysis of Lightweight CNNs for Resource-Constrained Devices: Predictive Performance, Efficiency Trade-offs, and Initialization Effects](https://mlsanity.com/paper/2505.03303/) (2025)
- [Vision Backbone Efficient Selection for Image Classification in Low-Data Regimes](https://mlsanity.com/paper/2410.08592/) (2024)
- [A Large-Scale Study on the Accuracy vs Cost Trade-offs of Training and Evaluation Settings in Fine-Grained Image Recognition](https://mlsanity.com/paper/2605.18700/) (2026)
- [Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency](https://mlsanity.com/paper/2607.01984/) (2026)
- [LIMODENet: Attention-Free Compact Encoders for Information-Preserving Onboard Satellite Image Restoration](https://mlsanity.com/paper/2609.14690/) (2026)
- [Robust Explanatory Analysis for Structured Optimization Networks (REASON) in Learning](https://mlsanity.com/paper/2610.19378/) (2026)

## Abstract (arXiv)

In contemporary computer vision applications, particularly image classification, architectural backbones pre-trained on large datasets like ImageNet are commonly employed as feature extractors. Despite the widespread use of these pre-trained convolutional neural networks (CNNs), there remains a gap in understanding the performance of various resource-efficient backbones across diverse domains and dataset sizes. Our study systematically evaluates multiple lightweight, pre-trained CNN backbones under consistent training settings across a variety of datasets, including natural images, medical images, galaxy images, and remote sensing images. This comprehensive analysis aims to aid machine learning practitioners in selecting the most suitable backbone for their specific problem, especially in scenarios involving small datasets where fine-tuning a pre-trained network is crucial. Even though attention-based architectures are gaining popularity, we observed that they tend to perform poorly under low data finetuning tasks compared to CNNs. We also observed that some CNN architectures such as ConvNeXt, RegNet and EfficientNet performs well compared to others on a diverse set of domains consistently. Our findings provide actionable insights into the performance trade-offs and effectiveness of different backbones, facilitating informed decision-making in model selection for a broad spectrum of computer vision domains. Our code is available here: https://github.com/pranavphoenix/Backbones