mlsanityData to 7 Oct 2026

Highway Networks

arXiv 1505.00387 ↗First posted 3 May 2015cs.LG cs.NEcited by 257

Not on the map. Why?

Abstract (arXiv)

There is plenty of theoretical and empirical evidence that depth of neural networks is a crucial ingredient for their success. However, network training becomes more difficult with increasing depth and training of very deep networks remains an open problem. In this extended abstract, we introduce a new architecture designed to ease gradient-based training of very deep networks. We refer to networks with this architecture as highway networks, since they allow unimpeded information flow across several layers on "information highways". The architecture is characterized by the use of gating units which learn to regulate the flow of information through a network. Highway networks with hundreds of layers can be trained directly using stochastic gradient descent and with a variety of activation functions, opening up the possibility of studying extremely deep and efficient architectures.

Key facts. Cited by 257 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 May 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 72; 2022: 50; 2023: 36; 2024: 26; 2025: 29; 2026: 44.

257 papers.

Sort:

By area:

  1. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  2. 2021Going deeper with Image Transformerscited by 759
  3. 2021Pay Attention to MLPscited by 367
  4. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  5. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  6. 2021TransFuse: Fusing Transformers and CNNs for Medical Image Segmentationcited by 258
  7. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  8. 2021SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effectscited by 103
  9. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  10. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  11. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  12. 2023InceptionNeXt: When Inception Meets ConvNeXtcited by 84
38 more
  1. 2021When and How Mixup Improves Calibrationcited by 59
  2. 2026Attention Residualscited by 57
  3. 2024Faster Diffusion via Temporal Attention Decompositioncited by 54
  4. 2021Tensor Methods in Computer Vision and Deep Learningcited by 52
  5. 2022Scalable Spatiotemporal Graph Neural Networkscited by 52
  6. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50
  7. 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 48
  8. 2021RegNet: Self-Regulated Network for Image Classificationcited by 44
  9. 2022An efficient encoder-decoder architecture with top-down attention for speech separationcited by 38
  10. 2021Boosting the Speed of Entity Alignment 10*: Dual Attention Matching Network with Normalized Hard Sample Miningcited by 37
  11. 2022FCN-Transformer Feature Fusion for Polyp Segmentationcited by 37
  12. 2021Modeling Fine-Grained Entity Types with Box Embeddingscited by 36
  13. 2024On the Scalability of GNNs for Molecular Graphscited by 36
  14. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  15. 2021Open-world Machine Learning: Applications, Challenges, and Opportunitiescited by 34
  16. 2022End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinkingcited by 33
  17. 2021Vision Transformers with Hierarchical Attentioncited by 31
  18. 2023Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matchingcited by 30
  19. 2021RAGA: Relation-aware Graph Attention Networks for Global Entity Alignmentcited by 29
  20. 2023Plasma Surrogate Modelling using Fourier Neural Operatorscited by 29
  21. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  22. 2021Alzheimer's Dementia Recognition Using Acoustic, Lexical, Disfluency and Speech Pause Features Robust to Noisy Inputscited by 27
  23. 2021Why don't people use character-level machine translation?cited by 27
  24. 2022ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generationcited by 26
  25. 2023ResFields: Residual Neural Fields for Spatiotemporal Signalscited by 26
  26. 2021OmniNet: Omnidirectional Representations from Transformerscited by 23
  27. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  28. 2021Rethinking Skip Connection with Layer Normalization in Transformers and ResNetscited by 22
  29. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  30. 2021Idiomatic Expression Identification using Semantic Compatibilitycited by 19
  31. 2022The Spectral Bias of Polynomial Neural Networkscited by 19
  32. 2023Learning to Skip for Language Modelingcited by 19
  33. 2024ResLoRA: Identity Residual Mapping in Low-Rank Adaptioncited by 18
  34. 2022Astronomia ex machina: a history, primer, and outlook on neural networks in astronomycited by 17
  35. 2021Multi-modal fusion with gating using audio, lexical and disfluency features for Alzheimer's Dementia recognition from spontaneous speechcited by 16
  36. 2021Task-agnostic Continual Learning with Hybrid Probabilistic Modelscited by 16
  37. 2022Exploiting Neighbor Effect: Conv-Agnostic GNNs Framework for Graphs with Heterophilycited by 15
  38. 2022Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rulescited by 15

50 most cited of 257 All 257 (JSON)