Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity
Not on the map. Why?
Abstract (arXiv)
The prefill stage of large language model (LLM) inference is a key computational bottleneck for long-context workloads. At short-to-moderate context lengths (1K--16K tokens), Feed-Forward Networks (FFNs) dominate this cost, accounting for most of the total FLOPs. Existing FFN sparsification methods, designed for autoregressive decoding, fail to exploit the prefill stage's parallelism and often degrade accuracy. To address this, we introduce FastForward, a predictive sparsity framework that accelerates LLM prefill through block-wise, context-aware FFN sparsity. FastForward combines (1) a lightweight expert predictor to select high-importance neurons per block, (2) an error compensation network to correct sparsity-induced errors, and (3) a layer-wise sparsity scheduler to allocate compute based on token-mixing importance. Across LLaMA and Qwen models up to 8B parameters, FastForward delivers up to 1.45× compute-bound speedup at 50% FFN sparsity with < 6% accuracy loss compared to the dense baseline on LongBench, substantially reducing Time-to-First-Token (TTFT) for efficient, long-context LLM inference on constrained hardware.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 30 Jan 2026.
Builds on
25 of 31 references.
References · 25
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
- 2021Measuring Coding Challenge Competence With APPScited by 879
- 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
- 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
- 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
- 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
- 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 129
- 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
- 2025Why do LLMs attend to the first token?cited by 122
13 more
- 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
- 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
- 2024Fast On-device LLM Inference with NPUscited by 58
- 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
- 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
- 2024TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attentioncited by 28
- 2024Selective Attention Improves Transformercited by 26
- 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
- 2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 20
- 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 18
- 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 18
- 2025AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inferencecited by 12
- 2025Adaptive Rank Allocation: Speeding Up Modern Transformers with RaNA Adapterscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.