mlsanityData to 7 Oct 2026

Training Very Deep Networks

arXiv 1507.06228 ↗First posted 22 Jul 2015cs.LG cs.NEcited by 172

Not on the map. Why?

Abstract (arXiv)

Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.

Key facts. Cited by 172 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jul 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 47; 2022: 28; 2023: 28; 2024: 24; 2025: 14; 2026: 31.

172 papers.

Sort:

By area:

  1. 2024Depth Anything V2cited by 1,385
  2. 2021Going deeper with Image Transformerscited by 759
  3. 2021Ensemble deep learning: A reviewcited by 283
  4. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  5. 2022Block-Recurrent Transformerscited by 107
  6. 2025TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learningcited by 107
  7. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  8. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 76
  9. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  10. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  11. 2021A Survey of Complex-Valued Neural Networkscited by 61
  12. 2023Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Modelscited by 59
38 more
  1. 2021MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motionscited by 54
  2. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  3. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 50
  4. 2021HINT: Hierarchical Interaction Network for Trial Outcome Prediction Leveraging Web Datacited by 49
  5. 2021Generate, Prune, Select: A Pipeline for Counterspeech Generation against Online Hate Speechcited by 42
  6. 2022A Law of Data Separation in Deep Learningcited by 36
  7. 2021Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safetycited by 35
  8. 2021Recomposing the Reinforcement Learning Building Blocks with Hypernetworkscited by 33
  9. 2022A Survey on Neural Open Information Extraction: Current Status and Future Directionscited by 27
  10. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
  11. 2021Selfish Sparse RNN Trainingcited by 26
  12. 2021Single Model Deep Learning on Imbalanced Small Datasets for Skin Lesion Classificationcited by 23
  13. 2021Parallelizing Legendre Memory Unit Trainingcited by 22
  14. 2023Not all layers are equally as important: Every Layer Counts BERTcited by 21
  15. 2024Uncertainty Quantification on Clinical Trial Outcome Predictioncited by 19
  16. 2022Refined Semantic Enhancement towards Frequency Diffusion for Video Captioningcited by 18
  17. 2024Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoningcited by 18
  18. 2021Towards Accurate and Compact Architectures via Neural Architecture Transformercited by 17
  19. 2021Learning to Fit Morphable Modelscited by 16
  20. 2022B2T Connection: Serving Stability and Performance in Deep Transformerscited by 16
  21. 2022A survey on attention mechanisms for medical applications: are we moving towards better algorithms?cited by 15
  22. 2021Alignment Attention by Matching Key and Query Distributionscited by 14
  23. 2023Computation-efficient Deep Learning for Computer Vision: A Surveycited by 14
  24. 2021Activation function design for deep networks: linearity and effective initialisationcited by 13
  25. 2021Why Adversarial Reprogramming Works, When It Fails, and How to Tell the Differencecited by 13
  26. 2023When to Learn What: Model-Adaptive Data Augmentation Curriculumcited by 13
  27. 2024Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequencescited by 13
  28. 2026LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Modelscited by 13
  29. 2021Semantic-Based Explainable AI: Leveraging Semantic Scene Graphs and Pairwise Ranking to Explain Robot Failurescited by 12
  30. 2021Semi-Supervised Semantic Segmentation of Vessel Images using Leaking Perturbationscited by 12
  31. 2024Residual Connections Harm Generative Representation Learningcited by 10
  32. 2024Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Surveycited by 10
  33. 2024Strengthening Layer Interaction via Dynamic Layer Attentioncited by 10
  34. 2026KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matricescited by 10
  35. 2024Neural Residual Diffusion Models for Deep Scalable Vision Generationcited by 9
  36. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  37. 2026Residual Context Diffusion Language Modelscited by 9
  38. 2021Text Summarization with Latent Queriescited by 8

50 most cited of 172 All 172 (JSON)