# MERCURY: Accelerating DNN Training By Exploiting Input Similarity

arXiv: https://arxiv.org/abs/2110.14904 · HTML: https://mlsanity.com/paper/2110.14904/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 68 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects](https://mlsanity.com/paper/2004.02806/) (2020)
- [Recent Advances in Convolutional Neural Networks](https://mlsanity.com/paper/1512.07108/) (2015)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning](https://mlsanity.com/paper/2012.10630/) (2020)
- [Trained Ternary Quantization](https://mlsanity.com/paper/1612.01064/) (2016)
- [SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks](https://mlsanity.com/paper/1708.04485/) (2017)
- [Mixed Precision Training of Convolutional Neural Networks using Integer Operations](https://mlsanity.com/paper/1802.00930/) (2018)
- [Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks](https://mlsanity.com/paper/1711.02213/) (2017)
- [Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools](https://mlsanity.com/paper/1903.11314/) (2019)
- [Learning What Data to Learn](https://mlsanity.com/paper/1702.08635/) (2017)
- [UCNN: Exploiting Computational Reuse in Deep Neural Networks via Weight Repetition](https://mlsanity.com/paper/1804.06508/) (2018)
- [Dynamic Sparse Graph for Efficient Deep Learning](https://mlsanity.com/paper/1810.00859/) (2018)
- [Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Training](https://mlsanity.com/paper/2009.10976/) (2020)
- [TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference](https://mlsanity.com/paper/2009.00748/) (2020)
- [Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training](https://mlsanity.com/paper/1805.08899/) (2018)

## Built on by (most cited first, 3 of 3)

- [Ditto: Accelerating Diffusion Model via Temporal Value Similarity](https://mlsanity.com/paper/2501.11211/) (2025)
- [ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction](https://mlsanity.com/paper/2305.13236/) (2023)
- [Leveraging KV Similarity for Online Structured Pruning in LLMs](https://mlsanity.com/paper/2512.07090/) (2025)

## Abstract (arXiv)

Deep Neural Networks (DNN) are computationally intensive to train. It consists of a large number of multidimensional dot products between many weights and input vectors. However, there can be significant similarity among input vectors. If one input vector is similar to another, its computations with the weights are similar to those of the other and, therefore, can be skipped by reusing the already-computed results. We propose a novel scheme, called MERCURY, to exploit input similarity during DNN training in a hardware accelerator. MERCURY uses Random Projection with Quantization (RPQ) to convert an input vector to a bit sequence, called Signature. A cache (MCACHE) stores signatures of recent input vectors along with the computed results. If the Signature of a new input vector matches that of an already existing vector in the MCACHE, the two vectors are found to have similarities. Therefore, the already-computed result is reused for the new vector. To the best of our knowledge, MERCURY is the first work that exploits input similarity using RPQ for accelerating DNN training in hardware. The paper presents a detailed design, workflow, and implementation of the MERCURY. Our experimental evaluation with twelve different deep learning models shows that MERCURY saves a significant number of computations and speeds up the model training by an average of 1.97X with an accuracy similar to the baseline system.