mlsanityData to 7 Oct 2026

Deep Networks with Stochastic Depth

arXiv 1603.09382 ↗First posted 30 Mar 2016cs.LG cs.CV cs.NEcited by 973

Not on the map. Why?

Abstract (arXiv)

Very deep convolutional networks with hundreds of layers have led to significant reductions in error on competitive benchmarks. Although the unmatched expressiveness of the many layers can be highly desirable at test time, training very deep networks comes with its own set of challenges. The gradients can vanish, the forward flow often diminishes, and the training time can be painfully slow. To address these problems, we propose stochastic depth, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time. We start with very deep networks but during training, for each mini-batch, randomly drop a subset of layers and bypass them with the identity function. This simple approach complements the recent success of residual networks. It reduces training time substantially and improves the test error significantly on almost all data sets that we used for evaluation. With stochastic depth we can increase the depth of residual networks even beyond 1200 layers and still yield meaningful improvements in test error (4.91% on CIFAR-10).

Key facts. Cited by 973 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 190; 2022: 243; 2023: 192; 2024: 133; 2025: 106; 2026: 108.

973 papers.

Sort:

By area:

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2023Segment Anythingcited by 8,158
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  12. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
38 more
  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  3. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  4. 2021Video Swin Transformercited by 1,090
  5. 2021Multiscale Vision Transformerscited by 896
  6. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  7. 2022A Generalist Agentcited by 802
  8. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  9. 2021Going deeper with Image Transformerscited by 759
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  12. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  13. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  14. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  15. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  16. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  17. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  18. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  19. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  20. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  21. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  22. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  23. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  24. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  25. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  26. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  27. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  28. 2021TransReID: Transformer-based Object Re-Identificationcited by 432
  29. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  30. 2022DeiT III: Revenge of the ViTcited by 417
  31. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  32. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  33. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  34. 2021Pay Attention to MLPscited by 367
  35. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  36. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  37. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  38. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345

50 most cited of 973 All 973 (JSON)