mlsanityData to 7 Oct 2026

IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs

arXiv 2405.02842 ↗First posted 5 May 2024cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

One limitation of existing Transformer-based models is that they cannot handle very long sequences as input since their self-attention operations exhibit quadratic time and space complexity. This problem becomes especially acute when Transformers are deployed on hardware platforms equipped only with CPUs. To address this issue, we propose a novel method for accelerating self-attention at inference time that works with pretrained Transformer models out-of-the-box without requiring retraining. We experiment using our method to accelerate various long-sequence Transformers, including a leading LLaMA 2-based LLM, on various benchmarks and demonstrate a greater speedup of 2.73x - 7.63x while retaining 98.6% - 99.6% of the accuracy of the original pretrained models. The code is available on our project website at https://yuzhenmao.github.io/IceFormer/.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 5 May 2024.

Builds on

22 of 30 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2020Reformer: The Efficient Transformercited by 1,658
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  10. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  11. 2021Random Feature Attentioncited by 350
  12. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
10 more
  1. 2018ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithmscited by 130
  2. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  3. 2015Practical and Optimal LSH for Angular Distancecited by 105
  4. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
  5. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  6. 2021H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequencescited by 47
  7. 2022Linear Complexity Randomized Self-attention Mechanismcited by 31
  8. 2014On Symmetric and Asymmetric LSHs for Inner Product Searchcited by 23
  9. 2021Optimizing Inference Performance of Transformers on CPUscited by 7
  10. 2017Fast k-Nearest Neighbour Search via Prioritized DCIcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 4.

8 papers.

  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  2. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  3. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  4. 2025ZETA: Leveraging Z-order Curves for Efficient Top-k Attentioncited by 9
  5. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMscited by 4
  6. 2026k-Maximum Inner Product Attention for Graph Transformers and the Expressive Power of GraphGPScited by 0
  7. 2026ARM: Attention with Routed-Memory for Learnable Sparse Controlcited by 0
  8. 2026DimPO: Dimensionality Reduction for Attention using Preference Optimizationcited by 0