# Attention, Distillation, and Tabularization: Towards Practical Neural Network-Based Prefetching

arXiv: https://arxiv.org/abs/2401.06362 · HTML: https://mlsanity.com/paper/2401.06362/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 54 references that are in the map.
- First posted on arXiv 23 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [DeepShift: Towards Multiplication-Less Neural Networks](https://mlsanity.com/paper/1905.13298/) (2019)
- [ShiftAddNet: A Hardware-Inspired Deep Network](https://mlsanity.com/paper/2010.12785/) (2020)
- [Multiplying Matrices Without Multiplying](https://mlsanity.com/paper/2106.10860/) (2021)
- [Learning Memory Access Patterns](https://mlsanity.com/paper/1803.02329/) (2018)
- [Accelerating CNN inference on FPGAs: A Survey](https://mlsanity.com/paper/1806.01683/) (2018)
- [The Championship Simulator: Architectural Simulation for Education and Competition](https://mlsanity.com/paper/2210.14324/) (2022)
- [GenGNN: A Generic FPGA Framework for Graph Neural Network Acceleration](https://mlsanity.com/paper/2201.08475/) (2022)
- [Fine-Grained Address Segmentation for Attention-Based Variable-Degree Prefetching](https://mlsanity.com/paper/2205.02269/) (2022)
- [Puppeteer: A Random Forest-based Manager for Hardware Prefetchers across the Memory Hierarchy](https://mlsanity.com/paper/2201.12027/) (2022)
- [Phases, Modalities, Temporal and Spatial Locality: Domain Specific ML Prefetcher for Accelerating Graph Analytics](https://mlsanity.com/paper/2212.05250/) (2022)
- [Lightweight ML-based Runtime Prefetcher Selection on Many-core Platforms](https://mlsanity.com/paper/2307.08635/) (2023)

## Built on by (most cited first, 1 of 1)

- [Learning Semantics, Not Addresses: Runtime Neural Prefetching for Far Memory](https://mlsanity.com/paper/2506.00384/) (2025)

## Abstract (arXiv)

Attention-based Neural Networks (NN) have demonstrated their effectiveness in accurate memory access prediction, an essential step in data prefetching. However, the substantial computational overheads associated with these models result in high inference latency, limiting their feasibility as practical prefetchers. To close the gap, we propose a new approach based on tabularization that significantly reduces model complexity and inference latency without sacrificing prediction accuracy. Our novel tabularization methodology takes as input a distilled, yet highly accurate attention-based model for memory access prediction and efficiently converts its expensive matrix multiplications into a hierarchy of fast table lookups. As an exemplar of the above approach, we develop DART, a prefetcher comprised of a simple hierarchy of tables. With a modest 0.09 drop in F1-score, DART reduces 99.99% of arithmetic operations from the large attention-based model and 91.83% from the distilled model. DART accelerates the large model inference by 170x and the distilled model by 9.4x. DART has comparable latency and storage costs as state-of-the-art rule-based prefetcher BO but surpasses it by 6.1% in IPC improvement. DART outperforms state-of-the-art NN-based prefetchers TransFetch by 33.1% and Voyager by 37.2% in terms of IPC improvement, primarily due to its low prefetching latency.