mlsanityData to 14 Oct 2026

AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Design

arXiv 2505.03745 ↗First posted 7 Apr 2025cs.AR cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Recently, large language models (LLMs) have achieved huge success in the natural language processing (NLP) field, driving a growing demand to extend their deployment from the cloud to edge devices. However, deploying LLMs on resource-constrained edge devices poses significant challenges, including (1) intensive computations and huge model sizes, (2) great memory and bandwidth demands introduced by the autoregressive generation process, and (3) limited scalability for handling long sequences. To address these challenges, we propose AccLLM, a comprehensive acceleration framework that enables efficient and fast long-context LLM inference through algorithm and hardware co-design. At the algorithmic level, we integrate (1) pruning, (2) Lambda-shaped attention, and (3) an innovative W2A8KV4 (2-bit weights, 8-bit activations, and 4-bit KV cache) quantization scheme, thus effectively reducing memory and bandwidth requirements while facilitating LLMs' long-sequence generation. At the hardware level, we design a dedicated FPGA-based accelerator with a reconfigurable computing engine to effectively and flexibly accommodate diverse operations arising from our compression algorithm, thereby fully translating the algorithmic innovations into tangible hardware efficiency. We validate AccLLM on the Xilinx Alveo U280 FPGA, demonstrating a 4.07x energy efficiency and a 2.98x throughput compared to the state-of-the-art work FlightLLM.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 58 of its 72 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

58 of 72 references.

References · 58

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  12. 2023Mistral 7Bcited by 3,164
38 more
  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  11. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  12. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  13. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  14. 2023A Comprehensive Overview of Large Language Modelscited by 648
  15. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  16. 2022Efficiently Scaling Transformer Inferencecited by 539
  17. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  18. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  19. 2021MetaICL: Learning to Learn In Contextcited by 454
  20. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  21. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  22. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  23. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  24. 2021Accelerating Sparse Deep Neural Networkscited by 223
  25. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  26. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  27. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  28. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  29. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  30. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  31. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  32. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 64
  33. 2016Pruning Convolutional Neural Networks for Resource Efficient Inferencecited by 48
  34. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 48
  35. 2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 45
  36. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  37. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 39
  38. 2022ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attentioncited by 29

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  2. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  3. 2023Efficient LLM Inference on CPUs
  4. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  5. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency
  6. 2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems
  7. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs
  8. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory