mlsanityData to 7 Oct 2026

Designing Network Design Spaces

arXiv 2003.13678 ↗First posted 30 Mar 2020cs.CV cs.LGcited by 954

Not on the map. Why?

Cited by 29 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we present a new network design paradigm. Our goal is to help advance the understanding of network design and discover design principles that generalize across settings. Instead of focusing on designing individual network instances, we design network design spaces that parametrize populations of networks. The overall process is analogous to classic manual design of networks, but elevated to the design space level. Using our methodology we explore the structure aspect of network design and arrive at a low-dimensional design space consisting of simple, regular networks that we call RegNet. The core insight of the RegNet parametrization is surprisingly simple: widths and depths of good networks can be explained by a quantized linear function. We analyze the RegNet design space and arrive at interesting findings that do not match the current practice of network design. The RegNet design space provides simple and fast networks that work well across a wide range of flop regimes. Under comparable training settings and flops, the RegNet models outperform the popular EfficientNet models while being up to 5x faster on GPUs.

Key facts. Cited by 954 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 202; 2022: 222; 2023: 182; 2024: 162; 2025: 109; 2026: 77.

954 papers.

Sort:

By area:

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2022Scalable Diffusion Models with Transformerscited by 5,931
  4. 2022A ConvNet for the 2020scited by 3,545
  5. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  6. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  7. 2021Transformers in Vision: A Surveycited by 1,009
  8. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  9. 2024VMamba: Visual State Space Modelcited by 955
  10. 2021Multiscale Vision Transformerscited by 896
  11. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  12. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
38 more
  1. 2021Going deeper with Image Transformerscited by 759
  2. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  5. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  6. 2021Transformer in Transformercited by 632
  7. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  8. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  9. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 520
  10. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  11. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  12. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  13. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  14. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  15. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  16. 2021Intriguing Properties of Vision Transformerscited by 446
  17. 2022DeiT III: Revenge of the ViTcited by 417
  18. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  19. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  20. 2021Early Convolutions Help Transformers See Bettercited by 395
  21. 2021Pay Attention to MLPscited by 367
  22. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  23. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  24. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  25. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  26. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  27. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  28. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  29. 2022VOS: Learning What You Don't Know by Virtual Outlier Synthesiscited by 300
  30. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  31. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  32. 2021Global Filter Networks for Image Classificationcited by 257
  33. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  34. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  35. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  36. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  37. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  38. 2021WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognitioncited by 221

50 most cited of 954 All 954 (JSON)