# LookupFFN: Making Transformers Compute-lite for CPU inference arXiv: https://arxiv.org/abs/2403.07221 ยท HTML: https://mlsanity.com/paper/2403.07221/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 50 references that are in the map. - First posted on arXiv 12 Mar 2024. ## Builds on (most cited first, 30 of 30) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017) - [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021) - [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017) - [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020) - [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020) - [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020) - [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017) - [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017) - [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018) - [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021) - [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019) - [Practical and Optimal LSH for Angular Distance](https://mlsanity.com/paper/1509.02897/) (2015) - [Network Pruning via Transformable Architecture Search](https://mlsanity.com/paper/1905.09717/) (2019) - [Fastfood: Approximate Kernel Expansions in Loglinear Time](https://mlsanity.com/paper/1408.3060/) (2014) - [SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems](https://mlsanity.com/paper/1903.03129/) (2019) - [Hashing-Based-Estimators for Kernel Density in High Dimensions](https://mlsanity.com/paper/1808.10530/) (2018) - [Optimizing CNN Model Inference on CPUs](https://mlsanity.com/paper/1809.02697/) (2018) - [Deep Fried Convnets](https://mlsanity.com/paper/1412.7149/) (2014) - [ACDC: A Structured Efficient Linear Layer](https://mlsanity.com/paper/1511.05946/) (2015) - [You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling](https://mlsanity.com/paper/2111.09714/) (2021) - [A New Unbiased and Efficient Class of LSH-Based Samplers and Estimators for Partition Function Computation in Log-Linear Models](https://mlsanity.com/paper/1703.05160/) (2017) - [Anatomy Of High-Performance Deep Learning Convolutions On SIMD Architectures](https://mlsanity.com/paper/1808.05567/) (2018) - [Multi Resolution Analysis (MRA) for Approximate Self-Attention](https://mlsanity.com/paper/2207.10284/) (2022) ## Built on by (most cited first, 8 of 8) - [MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling](https://mlsanity.com/paper/2602.03359/) (2026) - [MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter](https://mlsanity.com/paper/2406.04984/) (2024) - [MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers](https://mlsanity.com/paper/2411.12992/) (2024) - [Spark Transformer: Reactivating Sparsity in FFN and Attention](https://mlsanity.com/paper/2506.06644/) (2025) - [Composing Linear Layers from Irreducibles](https://mlsanity.com/paper/2507.11688/) (2025) - [SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV](https://mlsanity.com/paper/2509.25853/) (2025) - [Parallax: Runtime Parallelization for Operator Fallbacks in Heterogeneous Edge Systems](https://mlsanity.com/paper/2512.11532/) (2025) - [LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference](https://mlsanity.com/paper/2609.03079/) (2026) ## Abstract (arXiv) While GPU clusters are the de facto choice for training large deep neural network (DNN) models today, several reasons including ease of workflow, security and cost have led to efforts investigating whether CPUs may be viable for inference in routine use in many sectors of the industry. But the imbalance between the compute capabilities of GPUs and CPUs is huge. Motivated by these considerations, we study a module which is a workhorse within modern DNN architectures, GEMM based Feed Forward Networks (FFNs), and assess the extent to which it can be made compute- (or FLOP-) lite. Specifically, we propose an alternative formulation (we call it LookupFFN) to GEMM based FFNs inspired by the recent studies of using Locality Sensitive Hashing (LSH) to approximate FFNs. Our formulation recasts most essential operations as a memory look-up, leveraging the trade-off between the two resources on any platform: compute and memory (since CPUs offer it in abundance). For RoBERTa language model pretraining, our formulation achieves similar performance compared to GEMM based FFNs, while dramatically reducing the required FLOP. Our development is complemented with a detailed hardware profiling of strategies that will maximize efficiency -- not just on contemporary hardware but on products that will be offered in the near/medium term future. Code is avaiable at urlhttps://github.com/mlpen/LookupFFN.