mlsanityData to 7 Oct 2026

Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units

arXiv 1603.05201 ↗First posted 16 Mar 2016cs.LG cs.CVcited by 86

Not on the map. Why?

Abstract (arXiv)

Recently, convolutional neural networks (CNNs) have been used as a powerful tool to solve many problems of machine learning and computer vision. In this paper, we aim to provide insight on the property of convolutional neural networks, as well as a generic method to improve the performance of many CNN architectures. Specifically, we first examine existing CNN models and observe an intriguing property that the filters in the lower layers form pairs (i.e., filters with opposite phase). Inspired by our observation, we propose a novel, simple yet effective activation scheme called concatenated ReLU (CRelu) and theoretically analyze its reconstruction property in CNNs. We integrate CRelu into several state-of-the-art CNN architectures and demonstrate improvement in their recognition performance on CIFAR-10/100 and ImageNet datasets with fewer trainable parameters. Our results suggest that better understanding of the properties of CNNs can lead to significant performance improvement with a simple modification.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 18; 2023: 17; 2024: 16; 2025: 9; 2026: 2.

86 papers.

Sort:

  1. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  2. 2021GAN Inversion: A Surveycited by 366
  3. 2021NeRV: Neural Representations for Videoscited by 255
  4. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  5. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  6. 2021Trustworthy AI: A Computational Perspectivecited by 93
  7. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 63
  8. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  9. 2021Learning with Hyperspherical Uniformitycited by 35
  10. 2024Learning Continually by Spectral Regularizationcited by 34
  11. 2022Constrained Monotonic Neural Networkscited by 31
  12. 2022How important are activation functions in regression and classification? A survey, performance comparison, and future directionscited by 29
38 more
  1. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  2. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  3. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  4. 2021Adaptive Rational Activations to Boost Deep Reinforcement Learningcited by 23
  5. 2022Most Activation Functions Can Win the Lottery Without Excessive Depthcited by 21
  6. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  7. 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
  8. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  9. 2021Invertible DenseNets with Concatenated LipSwishcited by 17
  10. 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16
  11. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learningcited by 16
  12. 2021Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learningcited by 14
  13. 2024Plastic Learning with Deep Fourier Featurescited by 14
  14. 2022Graph-adaptive Rectified Linear Unit for Graph Neural Networkscited by 12
  15. 2022AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimizationcited by 10
  16. 2023Are GATs Out of Balance?cited by 9
  17. 2022Piecewise Linear Neural Networks and Deep Learningcited by 7
  18. 2024DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticitycited by 7
  19. 2022Update Compression for Deep Neural Networks on the Edgecited by 6
  20. 2022Adversarial Parameter Attack on Deep Neural Networkscited by 6
  21. 2023Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networkscited by 6
  22. 2025Activation Function Design Sustains Plasticity in Continual Learningcited by 6
  23. 2021RMNet: Equivalently Removing Residual Connection from Networkscited by 5
  24. 2022Retinex Image Enhancement Based on Sequential Decomposition With a Plug-and-Play Frameworkcited by 5
  25. 2023Deep Diversity-Enhanced Feature Representation of Hyperspectral Imagescited by 5
  26. 2025Spectral Collapse Drives Loss of Plasticity in Deep Continual Learningcited by 5
  27. 2021Phase Collapse in Neural Networkscited by 4
  28. 2021Why KDAC? A general activation function for knowledge discoverycited by 4
  29. 2022FaceTopoNet: Facial Expression Recognition using Face Topology Learningcited by 4
  30. 2023Increasing biases can be more efficient than increasing weightscited by 4
  31. 2024Deep Exploration with PAC-Bayescited by 4
  32. 2025Experience Replay Addresses Loss of Plasticity in Continual Learningcited by 4
  33. 2021Channel redundancy and overlap in convolutional neural networks with channel-wise NNK graphscited by 3
  34. 2022Training Thinner and Deeper Neural Networks: Jumpstart Regularizationcited by 3
  35. 2022Combining Intra-Risk and Contagion Risk for Enterprise Bankruptcy Prediction Using Graph Neural Networkscited by 3
  36. 2022Pushing the Efficiency Limit Using Structured Sparse Convolutionscited by 3
  37. 2023Learnable Heterogeneous Convolution: Learning both topology and strengthcited by 3
  38. 2023Bringing regularized optimal transport to lightspeed: a splitting method adapted for GPUscited by 3

50 most cited of 86 All 86 (JSON)