# Why KDAC? A general activation function for knowledge discovery

arXiv: https://arxiv.org/abs/2111.13858 · HTML: https://mlsanity.com/paper/2111.13858/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 125 references that are in the map.
- First posted on arXiv 27 Nov 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [Residual Flows for Invertible Generative Modeling](https://mlsanity.com/paper/1906.02735/) (2019)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020)
- [A survey on modern trainable activation functions](https://mlsanity.com/paper/2005.00817/) (2020)
- [Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units](https://mlsanity.com/paper/1603.05201/) (2016)
- [Chinese NER Using Lattice LSTM](https://mlsanity.com/paper/1805.02023/) (2018)
- [Noisy Activation Functions](https://mlsanity.com/paper/1603.00391/) (2016)
- [Parametric Exponential Linear Unit for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1605.09332/) (2016)
- [Deep Learning with S-shaped Rectified Linear Activation Units](https://mlsanity.com/paper/1512.07030/) (2015)
- [xUnit: Learning a Spatial Activation Function for Efficient Image Restoration](https://mlsanity.com/paper/1711.06445/) (2017)
- [Named Entity Recognition in Electronic Health Records Using Transfer Learning Bootstrapped Neural Networks](https://mlsanity.com/paper/1901.01592/) (2019)
- [Semi-Supervised Disentangled Framework for Transferable Named Entity Recognition](https://mlsanity.com/paper/2012.11805/) (2020)
- [SkillNER: Mining and Mapping Soft Skills from any Text](https://mlsanity.com/paper/2101.11431/) (2021)
- [Weighted Sigmoid Gate Unit for an Activation Function of Deep Neural Network](https://mlsanity.com/paper/1810.01829/) (2018)
- [An Evaluation of Recent Neural Sequence Tagging Models in Turkish Named Entity Recognition](https://mlsanity.com/paper/2005.07692/) (2020)
- [ASTRAL: Adversarial Trained LSTM-CNN for Named Entity Recognition](https://mlsanity.com/paper/2009.01041/) (2020)
- [A hybrid deep-learning approach for complex biochemical named entity recognition](https://mlsanity.com/paper/2012.10824/) (2020)
- [Method and Dataset Entity Mining in Scientific Literature: A CNN + Bi-LSTM Model with Self-attention](https://mlsanity.com/paper/2010.13583/) (2020)

## Built on by (most cited first, 4 of 4)

- [Adaptive Activation Functions for Predictive Modeling with Sparse Experimental Data](https://mlsanity.com/paper/2402.05401/) (2024)
- [A Zipf's Law-based Text Generation Approach for Addressing Imbalance in Entity Extraction](https://mlsanity.com/paper/2205.12636/) (2022)
- [A new hazard event classification model via deep learning and multifractal](https://mlsanity.com/paper/2209.05263/) (2022)
- [Yes, DLGM! A novel hierarchical model for hazard classification](https://mlsanity.com/paper/2209.04576/) (2022)

## Abstract (arXiv)

Deep learning oriented named entity recognition (DNER) has gradually become the paradigm of knowledge discovery, which greatly promotes domain intelligence. However, the current activation function of DNER fails to treat gradient vanishing, no negative output or non-differentiable existence, which may impede knowledge exploration caused by the omission and incomplete representation of latent semantics. To break through the dilemma, we present a novel activation function termed KDAC. Detailly, KDAC is an aggregation function with multiple conversion modes. The backbone of the activation region is the interaction between exponent and linearity, and the both ends extend through adaptive linear divergence, which surmounts the obstacle of gradient vanishing and no negative output. Crucially, the non-differentiable points are alerted and eliminated by an approximate smoothing algorithm. KDAC has a series of brilliant properties, including nonlinear, stable near-linear transformation and derivative, as well as dynamic style, etc. We perform experiments based on BERT-BiLSTM-CNN-CRF model on six benchmark datasets containing different domain knowledge, such as Weibo, Clinical, E-commerce, Resume, HAZOP and People's daily. The evaluation results show that KDAC is advanced and effective, and can provide more generalized activation to stimulate the performance of DNER. We hope that KDAC can be exploited as a promising activation function to devote itself to the construction of knowledge.