mlsanityData to 14 Oct 2026

Mish: A Self Regularized Non-Monotonic Activation Function

arXiv 1908.08681 ↗First posted 23 Aug 2019cs.LG cs.CV cs.NEcited by 141

Not on the map. Why?

Abstract (arXiv)

We propose textitMish, a novel self-regularized non-monotonic activation function which can be mathematically defined as: f(x)=xtanh(softplus(x)). As activation functions play a crucial role in the performance and training dynamics in neural networks, we validated experimentally on several well-known benchmarks against the best combinations of architectures and activation functions. We also observe that data augmentation techniques have a favorable effect on benchmarks like ImageNet-1k and MS-COCO across multiple architectures. For example, Mish outperformed Leaky ReLU on YOLOv4 with a CSP-DarkNet-53 backbone on average precision (AP50val) by 2.1% in MS-COCO object detection and ReLU on ResNet-50 on ImageNet-1k in Top-1 accuracy by ≈1% while keeping all other network parameters and hyperparameters constant. Furthermore, we explore the mathematical formulation of Mish in relation with the Swish family of functions and propose an intuitive understanding on how the first derivative behavior may be acting as a regularizer helping the optimization of deep neural networks. Code is publicly available at https://github.com/digantamisra98/Mish.

Key facts. Cited by 141 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 36; 2022: 33; 2023: 30; 2024: 19; 2025: 13; 2026: 10.

141 papers.

Sort:

By area:

  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  3. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  4. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  5. 2022YOLOv6: A Single-Stage Object Detection Framework for Industrial Applicationscited by 275
  6. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  7. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 155
  8. 2023A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAScited by 151
  9. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  10. 2022Concept Bottleneck Model with Additional Unsupervised Conceptscited by 79
  11. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RLcited by 68
  12. 2024Amortized Planning with Large-Scale Transformers: A Case Study on Chesscited by 49
38 more
  1. 2023PDPP: Projected Diffusion for Procedure Planning in Instructional Videoscited by 36
  2. 2023Adaptive Online Replanning with Diffusion Modelscited by 34
  3. 2022Thermodynamics-informed graph neural networkscited by 33
  4. 2023CoNIC Challenge: Pushing the Frontiers of Nuclear Detection, Segmentation, Classification and Countingcited by 33
  5. 2021PP-YOLOv2: A Practical Object Detectorcited by 30
  6. 2021Low Curvature Activations Reduce Overfitting in Adversarial Trainingcited by 29
  7. 2022How important are activation functions in regression and classification? A survey, performance comparison, and future directionscited by 29
  8. 2022Training Your Sparse Neural Network Better with Any Maskcited by 26
  9. 2025Temporal Difference Flowscited by 24
  10. 2021A survey on computational spectral reconstruction methods from RGB to hyperspectral imagingcited by 20
  11. 2022blob loss: instance imbalance aware loss functions for semantic segmentationcited by 18
  12. 2022Astronomia ex machina: a history, primer, and outlook on neural networks in astronomycited by 17
  13. 2024Mastering Chess with a Transformer Modelcited by 15
  14. 2021Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learningcited by 14
  15. 2022AESPA: Accuracy Preserving Low-degree Polynomial Activation for Fast Private Inferencecited by 14
  16. 2021Activation Functions in Artificial Neural Networks: A Systematic Overviewcited by 13
  17. 2021Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networkscited by 13
  18. 2022Graph-adaptive Rectified Linear Unit for Graph Neural Networkscited by 12
  19. 2023HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsitycited by 12
  20. 2023Efficient Planning with Latent Diffusioncited by 12
  21. 2025Learning Massively Multitask World Models for Continuous Controlcited by 12
  22. 2025Diffusion Models in Simulation-Based Inference: A Tutorial Reviewcited by 11
  23. 2021PnP-3D: A Plug-and-Play for 3D Point Cloudscited by 10
  24. 2021SERF: Towards better training of deep neural networks using log-Softplus ERror activation Functioncited by 10
  25. 2022Continuous Human Action Recognition for Human-Machine Interaction: A Reviewcited by 10
  26. 2022MEDIAR: Harmony of Data-Centric and Model-Centric for Multi-Modality Microscopycited by 10
  27. 2023Because Every Sensor Is Unique, so Is Every Pair: Handling Dynamicity in Traffic Forecastingcited by 9
  28. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  29. 2021A Dual-branch Network for Infrared and Visible Image Fusioncited by 8
  30. 2021DeepOrder: Deep Learning for Test Case Prioritization in Continuous Integration Testingcited by 8
  31. 2024Diffusion Models as Optimizers for Efficient Planning in Offline RLcited by 8
  32. 2022Data-Driven Deep Learning Based Hybrid Beamforming for Aerial Massive MIMO-OFDM Systems with Implicit CSIcited by 7
  33. 2023Dual-Diffusion: Dual Conditional Denoising Diffusion Probabilistic Models for Blind Super-Resolution Reconstruction in RSIscited by 7
  34. 2024Research on Deep Learning Model of Feature Extraction Based on Convolutional Neural Networkcited by 7
  35. 2024Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemblecited by 7
  36. 2026Compositional Planning with Jumpy World Modelscited by 7
  37. 2021Rethinking Generic Camera Models for Deep Single Image Camera Calibration to Recover Rotation and Fisheye Distortioncited by 6
  38. 2022Team Yao at Factify 2022: Utilizing Pre-trained Models and Co-attention Networks for Multi-Modal Fact Verificationcited by 6

50 most cited of 141 All 141 (JSON)