# PaCKD: Pattern-Clustered Knowledge Distillation for Compressing Memory Access Prediction Models

arXiv: https://arxiv.org/abs/2402.13441 · HTML: https://mlsanity.com/paper/2402.13441/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [What Do Compressed Deep Neural Networks Forget?](https://mlsanity.com/paper/1911.05248/) (2019)
- [Learning Memory Access Patterns](https://mlsanity.com/paper/1803.02329/) (2018)
- [Sequencer: Deep LSTM for Image Classification](https://mlsanity.com/paper/2205.01972/) (2022)
- [The GAP Benchmark Suite](https://mlsanity.com/paper/1508.03619/) (2015)
- [Fine-Grained Address Segmentation for Attention-Based Variable-Degree Prefetching](https://mlsanity.com/paper/2205.02269/) (2022)
- [Accelerating PageRank using Partition-Centric Processing](https://mlsanity.com/paper/1709.07122/) (2017)

## Built on by (most cited first, 2 of 2)

- [AlzheimerRAG: Multimodal Retrieval Augmented Generation for Clinical Use Cases using PubMed articles](https://mlsanity.com/paper/2412.16701/) (2024)
- [TabConv: Low-Computation CNN Inference via Table Lookups](https://mlsanity.com/paper/2404.05872/) (2024)

## Abstract (arXiv)

Deep neural networks (DNNs) have proven to be effective models for accurate Memory Access Prediction (MAP), a critical task in mitigating memory latency through data prefetching. However, existing DNN-based MAP models suffer from the challenges such as significant physical storage space and poor inference latency, primarily due to their large number of parameters. These limitations render them impractical for deployment in real-world scenarios. In this paper, we propose PaCKD, a Pattern-Clustered Knowledge Distillation approach to compress MAP models while maintaining the prediction performance. The PaCKD approach encompasses three steps: clustering memory access sequences into distinct partitions involving similar patterns, training large pattern-specific teacher models for memory access prediction for each partition, and training a single lightweight student model by distilling the knowledge from the trained pattern-specific teachers. We evaluate our approach on LSTM, MLP-Mixer, and ResNet models, as they exhibit diverse structures and are widely used for image classification tasks in order to test their effectiveness in four widely used graph applications. Compared to the teacher models with 5.406M parameters and an F1-score of 0.4626, our student models achieve a 552× model size compression while maintaining an F1-score of 0.4538 (with a 1.92% performance drop). Our approach yields an 8.70% higher result compared to student models trained with standard knowledge distillation and an 8.88% higher result compared to student models trained without any form of knowledge distillation.