# Evaluating and Enhancing Robustness of Deep Recommendation Systems Against Hardware Errors

arXiv: https://arxiv.org/abs/2307.10244 · HTML: https://mlsanity.com/paper/2307.10244/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 60 references that are in the map.
- First posted on arXiv 17 Jul 2023.

## Builds on (most cited first, 24 of 24)

- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks](https://mlsanity.com/paper/1708.04617/) (2017)
- [The Architectural Implications of Facebook's DNN-based Personalized Recommendation](https://mlsanity.com/paper/1906.03109/) (2019)
- [Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems](https://mlsanity.com/paper/2003.05622/) (2020)
- [Understanding Training Efficiency of Deep Learning Recommendation Models at Scale](https://mlsanity.com/paper/2011.05497/) (2020)
- [FT-ClipAct: Resilience Analysis of Deep Neural Networks and Improving their Fault Tolerance using Clipped Activation](https://mlsanity.com/paper/1912.00941/) (2019)
- [Understanding Capacity-Driven Scale-Out Neural Recommendation Inference](https://mlsanity.com/paper/2011.02084/) (2020)
- [Analyzing and Mitigating the Impact of Permanent Faults on a Systolic Array Based Neural Network Accelerator](https://mlsanity.com/paper/1802.04657/) (2018)
- [Heterogeneous Acceleration Pipeline for Recommendation System Training](https://mlsanity.com/paper/2204.05436/) (2022)
- [Cross-Stack Workload Characterization of Deep Recommendation Systems](https://mlsanity.com/paper/2010.05037/) (2020)
- [RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance](https://mlsanity.com/paper/2105.08820/) (2021)
- [Detecting silent data corruptions in the wild](https://mlsanity.com/paper/2203.08989/) (2022)
- [Efficient Soft-Error Detection for Low-precision Deep Learning Recommendation Models](https://mlsanity.com/paper/2103.00130/) (2021)
- [MoRS: An Approximate Fault Modelling Framework for Reduced-Voltage SRAMs](https://mlsanity.com/paper/2110.05855/) (2021)
- [Understanding RowHammer Under Reduced Wordline Voltage: An Experimental Study Using Real DRAM Devices](https://mlsanity.com/paper/2206.09999/) (2022)
- [Thales: Formulating and Estimating Architectural Vulnerability Factors for DNN Accelerators](https://mlsanity.com/paper/2212.02649/) (2022)

## Built on by (most cited first, 1 of 1)

- [Understanding Silent Data Corruption in LLM Training](https://mlsanity.com/paper/2502.12340/) (2025)

## Abstract (arXiv)

Deep recommendation systems (DRS) heavily depend on specialized HPC hardware and accelerators to optimize energy, efficiency, and recommendation quality. Despite the growing number of hardware errors observed in large-scale fleet systems where DRS are deployed, the robustness of DRS has been largely overlooked. This paper presents the first systematic study of DRS robustness against hardware errors. We develop Terrorch, a user-friendly, efficient and flexible error injection framework on top of the widely-used PyTorch. We evaluate a wide range of models and datasets and observe that the DRS robustness against hardware errors is influenced by various factors from model parameters to input characteristics. We also explore 3 error mitigation methods including algorithm based fault tolerance (ABFT), activation clipping and selective bit protection (SBP). We find that applying activation clipping can recover up to 30% of the degraded AUC-ROC score, making it a promising mitigation method.