# TResNet: High Performance GPU-Dedicated Architecture

arXiv: https://arxiv.org/abs/2003.13630 · HTML: https://mlsanity.com/paper/2003.13630/

## Key facts

- Cited by 119 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2020.

## Built on by (most cited first, 30 of 119)

- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [Revisiting ResNets: Improved Training and Scaling Strategies](https://mlsanity.com/paper/2103.07579/) (2021)
- [Query2Label: A Simple Transformer Way to Multi-Label Classification](https://mlsanity.com/paper/2107.10834/) (2021)
- [RepViT: Revisiting Mobile CNN From ViT Perspective](https://mlsanity.com/paper/2307.09283/) (2023)
- [LambdaNetworks: Modeling Long-Range Interactions Without Attention](https://mlsanity.com/paper/2102.08602/) (2021)
- [DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics](https://mlsanity.com/paper/2210.02438/) (2022)
- [An Image is Worth 16x16 Words, What is a Video Worth?](https://mlsanity.com/paper/2103.13915/) (2021)
- [ML-Decoder: Scalable and Versatile Classification Head](https://mlsanity.com/paper/2111.12933/) (2021)
- [Residual Attention: A Simple but Effective Method for Multi-Label Recognition](https://mlsanity.com/paper/2108.02456/) (2021)
- [YOLO5Face: Why Reinventing a Face Detector](https://mlsanity.com/paper/2105.12931/) (2021)
- [AutoDropout: Learning Dropout Patterns to Regularize Deep Networks](https://mlsanity.com/paper/2101.01761/) (2021)
- [TkML-AP: Adversarial Attacks to Top-k Multi-Label Learning](https://mlsanity.com/paper/2108.00146/) (2021)
- [Multi-label Classification with Partial Annotations using Class-aware Selective Loss](https://mlsanity.com/paper/2110.10955/) (2021)
- [Semantic Diversity Learning for Zero-Shot Multi-label Classification](https://mlsanity.com/paper/2105.05926/) (2021)
- [sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification](https://mlsanity.com/paper/2108.10566/) (2021)
- [Searching for Fast Model Families on Datacenter Accelerators](https://mlsanity.com/paper/2102.05610/) (2021)
- [Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge](https://mlsanity.com/paper/2310.16112/) (2023)
- [MlTr: Multi-label Classification with Transformer](https://mlsanity.com/paper/2106.06195/) (2021)
- [HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search](https://mlsanity.com/paper/2102.11646/) (2021)
- [Large Batch Simulation for Deep Reinforcement Learning](https://mlsanity.com/paper/2103.07013/) (2021)
- [PARFormer: Transformer-based Multi-Task Network for Pedestrian Attribute Recognition](https://mlsanity.com/paper/2304.07230/) (2023)
- [Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and Benchmark](https://mlsanity.com/paper/2103.10895/) (2021)
- [PIDray: A Large-scale X-ray Benchmark for Real-World Prohibited Item Detection](https://mlsanity.com/paper/2211.10763/) (2022)
- [MobileMamba: Lightweight Multi-Receptive Visual Mamba Network](https://mlsanity.com/paper/2411.15941/) (2024)

## Abstract (arXiv)

Many deep learning models, developed in recent years, reach higher ImageNet accuracy than ResNet50, with fewer or comparable FLOPS count. While FLOPs are often seen as a proxy for network efficiency, when measuring actual GPU training and inference throughput, vanilla ResNet50 is usually significantly faster than its recent competitors, offering better throughput-accuracy trade-off. In this work, we introduce a series of architecture modifications that aim to boost neural networks' accuracy, while retaining their GPU training and inference efficiency. We first demonstrate and discuss the bottlenecks induced by FLOPs-optimizations. We then suggest alternative designs that better utilize GPU structure and assets. Finally, we introduce a new family of GPU-dedicated models, called TResNet, which achieve better accuracy and efficiency than previous ConvNets. Using a TResNet model, with similar GPU throughput to ResNet50, we reach 80.8 top-1 accuracy on ImageNet. Our TResNet models also transfer well and achieve state-of-the-art accuracy on competitive single-label classification datasets such as Stanford cars (96.0%), CIFAR-10 (99.0%), CIFAR-100 (91.5%) and Oxford-Flowers (99.1%). They also perform well on multi-label classification and object detection tasks. Implementation is available at: https://github.com/mrT23/TResNet.