# ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNs

arXiv: https://arxiv.org/abs/2406.06313 · HTML: https://mlsanity.com/paper/2406.06313/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 56 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [MCUNet: Tiny Deep Learning on IoT Devices](https://mlsanity.com/paper/2007.10319/) (2020)
- [Adversarially Robust Distillation](https://mlsanity.com/paper/1905.09747/) (2019)
- [FT-ClipAct: Resilience Analysis of Deep Neural Networks and Improving their Fault Tolerance using Clipped Activation](https://mlsanity.com/paper/1912.00941/) (2019)
- [A Systematic Literature Review on Hardware Reliability Assessment Methods for Deep Neural Networks](https://mlsanity.com/paper/2305.05750/) (2023)
- [FT-CNN: Algorithm-Based Fault Tolerance for Convolutional Neural Networks](https://mlsanity.com/paper/2003.12203/) (2020)
- [FAT: Training Neural Networks for Reliable Inference Under Hardware Faults](https://mlsanity.com/paper/2011.05873/) (2020)
- [FitAct: Error Resilient Deep Neural Networks via Fine-Grained Post-Trainable Activation Functions](https://mlsanity.com/paper/2112.13544/) (2021)
- [Statistical Perspectives on Reliability of Artificial Intelligence Systems](https://mlsanity.com/paper/2111.05391/) (2021)
- [Fast and Accurate Error Simulation for CNNs against Soft Errors](https://mlsanity.com/paper/2206.02051/) (2022)
- [DeepVigor: Vulnerability Value Ranges and Factors for DNNs' Reliability Assessment](https://mlsanity.com/paper/2303.06931/) (2023)
- [Enhancing Fault Resilience of QNNs by Selective Neuron Splitting](https://mlsanity.com/paper/2306.09973/) (2023)
- [HyCA: A Hybrid Computing Architecture for Fault Tolerant Deep Learning](https://mlsanity.com/paper/2106.04772/) (2021)
- [Fault-Aware Design and Training to Enhance DNNs Reliability with Zero-Overhead](https://mlsanity.com/paper/2205.14420/) (2022)
- [Cost-Effective Fault Tolerance for CNNs Using Parameter Vulnerability Based Hardening and Pruning](https://mlsanity.com/paper/2405.10658/) (2024)
- [FARe: Fault-Aware GNN Training on ReRAM-based PIM Accelerators](https://mlsanity.com/paper/2401.10522/) (2024)

## Built on by (most cited first, 1 of 1)

- [Enhancing Neural Network Robustness Against Fault Injection Through Non-linear Weight Transformations](https://mlsanity.com/paper/2411.19027/) (2024)

## Abstract (arXiv)

Deep Neural Networks (DNNs) are extensively employed in safety-critical applications where ensuring hardware reliability is a primary concern. To enhance the reliability of DNNs against hardware faults, activation restriction techniques significantly mitigate the fault effects at the DNN structure level, irrespective of accelerator architectures. State-of-the-art methods offer either neuron-wise or layer-wise clipping activation functions. They attempt to determine optimal clipping thresholds using heuristic and learning-based approaches. Layer-wise clipped activation functions cannot preserve DNNs resilience at high bit error rates. On the other hand, neuron-wise clipping activation functions introduce considerable memory overhead due to the addition of parameters, which increases their vulnerability to faults. Moreover, the heuristic-based optimization approach demands numerous fault injections during the search process, resulting in time-consuming threshold identification. On the other hand, learning-based techniques that train thresholds for entire layers concurrently often yield sub-optimal results. In this work, first, we demonstrate that it is not essential to incorporate neuron-wise activation functions throughout all layers in DNNs. Then, we propose a hybrid clipped activation function that integrates neuron-wise and layer-wise methods that apply neuron-wise clipping only in the last layer of DNNs. Additionally, to attain optimal thresholds in the clipping activation function, we introduce ProAct, a progressive training methodology. This approach iteratively trains the thresholds on a layer-by-layer basis, aiming to obtain optimal threshold values in each layer separately.