mlsanityData to 14 Oct 2026

DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs

arXiv 2403.19588 ↗First posted 28 Mar 2024cs.CV cs.LG cs.NEcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper revives Densely Connected Convolutional Networks (DenseNets) and reveals the underrated effectiveness over predominant ResNet-style architectures. We believe DenseNets' potential was overlooked due to untouched training methods and traditional design elements not fully revealing their capabilities. Our pilot study shows dense connections through concatenation are strong, demonstrating that DenseNets can be revitalized to compete with modern architectures. We methodically refine suboptimal components - architectural adjustments, block redesign, and improved training recipes towards widening DenseNets and boosting memory efficiency while keeping concatenation shortcuts. Our models, employing simple architectural elements, ultimately surpass Swin Transformer, ConvNeXt, and DeiT-III - key architectures in the residual learning lineage. Furthermore, our models exhibit near state-of-the-art performance on ImageNet-1K, competing with the very recent models and downstream tasks, ADE20k semantic segmentation, and COCO object detection/instance segmentation. Finally, we provide empirical analyses that uncover the merits of the concatenation over additive shortcuts, steering a renewed preference towards DenseNet-style designs. Our code is available at https://github.com/naver-ai/rdnet.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 85 of its 107 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

85 of 107 references.

References · 85

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2017Mask R-CNNcited by 7,277
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  12. 2016Layer Normalizationcited by 5,669
38 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2014Going Deeper with Convolutionscited by 4,241
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  8. 2022A ConvNet for the 2020scited by 3,579
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  13. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  14. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  15. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  16. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  17. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  18. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  19. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  20. 2019Natural Adversarial Examplescited by 1,408
  21. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  22. 2017Random Erasing Data Augmentationcited by 1,239
  23. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  24. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  25. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  26. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  27. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  28. 2016Deep Networks with Stochastic Depthcited by 979
  29. 2020Designing Network Design Spacescited by 960
  30. 2021Going deeper with Image Transformerscited by 764
  31. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  32. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  33. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  34. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  35. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  36. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  37. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429
  38. 2022DeiT III: Revenge of the ViTcited by 419

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 2.

12 papers.

  1. 2024reBEN: Refined BigEarthNet Dataset for Remote Sensing Image Analysiscited by 48
  2. 2025OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernelscited by 25
  3. 2025HCMA-UNet: A Hybrid CNN-Mamba UNet with Axial Self-Attention for Efficient Breast Cancer Segmentationcited by 4
  4. 2025A2Mamba: Attention-augmented State Space Models for Visual Recognitioncited by 1
  5. 2025NAT: Learning to Attack Neurons for Enhanced Adversarial Transferabilitycited by 1
  6. 2026Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routingcited by 1
  7. 2024ResidualDroppath: Enhancing Feature Reuse over Residual Connectionscited by 0
  8. 2025Spectral-Adaptive Modulation Networks for Visual Perceptioncited by 0
  9. 2025DBF-UNet: A Two-Stage Framework for Carotid Artery Segmentation with Pseudo-Label Generationcited by 0
  10. 2025InceptionMamba: An Efficient Hybrid Network with Large Band Convolution and Bottleneck Mambacited by 0
  11. 2025ASCMamba: Multimodal Time-Frequency Mamba for Acoustic Scene Classificationcited by 0
  12. 2026Isotropic Embedding Perturbations for Robust Vision Language Encoderscited by 0