mlsanityData to 7 Oct 2026

One weird trick for parallelizing convolutional neural networks

arXiv 1404.5997 ↗First posted 23 Apr 2014cs.NE cs.DC cs.LGcited by 264

Not on the map. Why?

Abstract (arXiv)

I present a new way to parallelize the training of convolutional neural networks across multiple GPUs. The method scales significantly better than all alternatives when applied to modern convolutional neural networks.

Key facts. Cited by 264 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Apr 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 54; 2022: 61; 2023: 61; 2024: 44; 2025: 28; 2026: 16.

264 papers.

Sort:

By area:

  1. 2022Visual Prompt Tuningcited by 1,674
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  3. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  4. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  5. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  6. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  7. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 115
  8. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  9. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 103
  10. 2022Denoising diffusion models for out-of-distribution detectioncited by 102
  11. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  12. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
38 more
  1. 2021ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shotcited by 86
  2. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  3. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 74
  4. 2022Human alignment of neural network representationscited by 74
  5. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  6. 2021On Large-Cohort Training for Federated Learningcited by 73
  7. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  8. 2023Separable Physics-Informed Neural Networkscited by 64
  9. 2023Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Modelscited by 61
  10. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  11. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  12. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  13. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  14. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  15. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  16. 2022NeRF-RPN: A general framework for object detection in NeRFscited by 46
  17. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  18. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  19. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  20. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  21. 2023Improving neural network representations using human similarity judgmentscited by 38
  22. 2022Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positivescited by 37
  23. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 36
  24. 2023The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenescited by 34
  25. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
  26. 2024LiGNN: Graph Neural Networks at LinkedIncited by 31
  27. 2024The Scene Language: Representing Scenes with Programs, Words, and Embeddingscited by 31
  28. 2022Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysiscited by 29
  29. 2021Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Trainingcited by 28
  30. 2023Why (and When) does Local SGD Generalize Better than SGD?cited by 26
  31. 2021Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Accelerationcited by 25
  32. 2021Local Calibration: Metrics and Recalibrationcited by 23
  33. 2023Topology-aware Federated Learning in Edge Computing: A Comprehensive Surveycited by 23
  34. 2025BroRL: Scaling Reinforcement Learning via Broadened Explorationcited by 23
  35. 2021Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learningcited by 22
  36. 2023How to Scale Your EMAcited by 22
  37. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  38. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22

50 most cited of 264 All 264 (JSON)