mlsanityData to 7 Oct 2026

Dynamic ReLU

arXiv 2003.10027 ↗First posted 22 Mar 2020cs.CVcited by 55

Not on the map. Why?

Abstract (arXiv)

Rectified linear units (ReLU) are commonly used in deep neural networks. So far ReLU and its generalizations (non-parametric or parametric) are static, performing identically for all input samples. In this paper, we propose dynamic ReLU (DY-ReLU), a dynamic rectifier of which parameters are generated by a hyper function over all in-put elements. The key insight is that DY-ReLU encodes the global context into the hyper function, and adapts the piecewise linear activation function accordingly. Compared to its static counterpart, DY-ReLU has negligible extra computational cost, but significantly more representation capability, especially for light-weight neural networks. By simply using DY-ReLU for MobileNetV2, the top-1 accuracy on ImageNet classification is boosted from 72.0% to 76.2% with only 5% additional FLOPs.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 7; 2023: 11; 2024: 13; 2025: 8; 2026: 2.

55 papers.

Sort:

By area:

  1. 2021Dynamic Neural Networks: A Surveycited by 405
  2. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  3. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  4. 2022MetaFormer Baselines for Visioncited by 131
  5. 2022MobileOne: An Improved One millisecond Mobile Backbonecited by 109
  6. 2021Lite-HRNet: A Lightweight High-Resolution Networkcited by 91
  7. 2024SHViT: Single-Head Vision Transformer with Memory Efficient Macro Designcited by 55
  8. 2024Rewrite the Starscited by 55
  9. 2021TAda! Temporally-Adaptive Convolutions for Video Understandingcited by 36
  10. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  11. 2023Dynamic Perceiver for Efficient Visual Recognitioncited by 33
  12. 2021MicroNet: Improving Image Recognition with Extremely Low FLOPscited by 26
38 more
  1. 2022DynamicISP: Dynamically Controlled Image Signal Processor for Image Recognitioncited by 23
  2. 2024A comprehensive survey of research towards AI-enabled unmanned aerial systems in pre-, active-, and post-wildfire managementcited by 23
  3. 2023Object Detection with Transformers: A Reviewcited by 22
  4. 2024Automate Strategy Finding with LLM in Quant Investmentcited by 21
  5. 2024EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Dualitycited by 17
  6. 2021Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learningcited by 14
  7. 2021Dynamic Binary Neural Network by learning channel-wise thresholdscited by 13
  8. 2021MutualNet: Adaptive ConvNet via Mutual Learning from Different Model Configurationscited by 12
  9. 2022Graph-adaptive Rectified Linear Unit for Graph Neural Networkscited by 12
  10. 2022MoCoViT: Mobile Convolutional Vision Transformercited by 11
  11. 2022Linear Leaky-Integrate-and-Fire Neuron Model Based Spiking Neural Networks and Its Mapping Relationship to Deep Neural Networkscited by 8
  12. 2025Hybrid-Balance GFlowNet for Solving Vehicle Routing Problemscited by 7
  13. 2023Temporally-Adaptive Models for Efficient Video Understandingcited by 6
  14. 2024MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolutioncited by 6
  15. 2024Trainable Highly-expressive Activation Functionscited by 6
  16. 2024Steganographic Passport: An Owner and User Verifiable Credential for Deep Model IP Protection Without Retrainingcited by 5
  17. 2021Accurate and Real-time 3D Pedestrian Detection Using an Efficient Attentive Pillar Networkcited by 4
  18. 2023Human-Inspired Facial Sketch Synthesis with Dynamic Adaptationcited by 4
  19. 2024FocSAM: Delving Deeply into Focused Objects in Segmenting Anythingcited by 4
  20. 2025CoDiff: Conditional Diffusion Model for Collaborative 3D Object Detectioncited by 4
  21. 2022Video Mobile-Former: Video Recognition with Efficient Global Spatial-temporal Modelingcited by 3
  22. 2023Boosting Convolutional Neural Networks with Middle Spectrum Grouped Convolutioncited by 3
  23. 2023Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Modelscited by 3
  24. 2021Sparta: Spatially Attentive and Adversarially Robust Activationcited by 2
  25. 2023Dynamic Mobile-Former: Strengthening Dynamic Convolution with Attention and Residual Connection in Kernel Spacecited by 2
  26. 2024SegHist: A General Segmentation-based Framework for Chinese Historical Document Text Line Detectioncited by 2
  27. 2025MID-L: Matrix-Interpolated Dropout Layer with Layer-wise Neuron Selectioncited by 2
  28. 2021Self-Supervised Deep Graph Embedding with High-Order Information Fusion for Community Discoverycited by 1
  29. 2023Dynamic Shuffle: An Efficient Channel Mixture Methodcited by 1
  30. 2023X-HRNet: Towards Lightweight Human Pose Estimation with Spatially Unidimensional Self-Attentioncited by 1
  31. 2023DyRA: Portable Dynamic Resolution Adjustment Network for Existing Detectorscited by 1
  32. 2025VeLU: Variance-enhanced Learning Unit for Deep Neural Networkscited by 1
  33. 2021Shift-and-Balance Attentioncited by 0
  34. 2021First Power Linear Unit with Signcited by 0
  35. 2024Multi-scale Unified Network for Image Classificationcited by 0
  36. 2024RepAct: The Re-parameterizable Adaptive Activation Functioncited by 0
  37. 2024Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruningcited by 0
  38. 2025HybridoNet-Adapt: A Domain-Adapted Framework for Accurate Lithium-Ion Battery RUL Predictioncited by 0

50 most cited of 55 All 55 (JSON)