Zero-Space Cost Fault Tolerance for Transformer-based Language Models on ReRAM
Not on the map. Why?
Abstract (arXiv)
Resistive Random Access Memory (ReRAM) has emerged as a promising platform for deep neural networks (DNNs) due to its support for parallel in-situ matrix-vector multiplication. However, hardware failures, such as stuck-at-fault defects, can result in significant prediction errors during model inference. While additional crossbars can be used to address these failures, they come with storage overhead and are not efficient in terms of space, energy, and cost. In this paper, we propose a fault protection mechanism that incurs zero space cost. Our approach includes: 1) differentiable structure pruning of rows and columns to reduce model redundancy, 2) weight duplication and voting for robust output, and 3) embedding duplicated most significant bits (MSBs) into the model weight. We evaluate our method on nine tasks of the GLUE benchmark with the BERT model, and experimental results prove its effectiveness.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 11 of its 27 references that are in the map. First posted on arXiv 22 Jan 2024.
Builds on
11 of 27 references.
References · 11
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
- 2018DARTS: Differentiable Architecture Searchcited by 1,782
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2016Training Sparse Neural Networkscited by 73
- 2017NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigmcited by 60
- 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
- 2022SWIM: Selective Write-Verify for Computing-in-Memory Neural Acceleratorscited by 16
- 2022Computing-In-Memory Neural Network Accelerators for Safety-Critical Systems: Can Small Device Variations Be Disastrous?cited by 12
- 2021FORMS: Fine-grained Polarized ReRAM-based In-situ Computation for Mixed-signal DNN Acceleratorcited by 10
- 2023Improving Realistic Worst-Case Performance of NVCiM DNN Accelerators through Training with Right-Censored Gaussian Noisecited by 8
Built on by
Citing papers per year: 2024: 2.
2 papers.