# Cramnet: Layer-wise Deep Neural Network Compression with Knowledge Transfer from a Teacher Network

arXiv: https://arxiv.org/abs/1904.05982 · HTML: https://mlsanity.com/paper/1904.05982/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 1 of 1)

- [Sample-based Dynamic Hierarchical Transformer with Layer and Head Flexibility via Contextual Bandit](https://mlsanity.com/paper/2312.03038/) (2023)

## Abstract (arXiv)

Neural Networks accomplish amazing things, but they suffer from computational and memory bottlenecks that restrict their usage. Nowhere can this be better seen than in the mobile space, where specialized hardware is being created just to satisfy the demand for neural networks. Previous studies have shown that neural networks have vastly more connections than they actually need to do their work. This thesis develops a method that can compress networks to less than 10% of memory and less than 25% of computational power, without loss of accuracy, and without creating sparse networks that require special code to run.