mlsanityData to 14 Oct 2026

FastTPS: An Optimized Method for LLM Token Phase for AI accelerators

arXiv 2607.11211 ↗First posted 13 Jul 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The popularity of large language models (LLMs) escalates an ongoing demand for effective inference. However, due to the sequential processing of tokens during the token phase in decoder-only LLMs inference, the inherent low parallelism leads to reduced throughput and suboptimal utilization of the computing units on artificial intelligence (AI) accelerators, particularly when handling long-sequence inputs that impose significant memory overhead. Recently, many reported methods have been developed as potential solutions, since they emerge with numeric deviation. This paper presents FastTPS, a high performance and low-precision loss method for accelerating the token-phase in LLM inference on general AI accelerators which includes three key components: (1) AI accelerator-enabled reloading-free KV Cache concatenation which decreases memory access overhead as well as enables full fusion of Attention, (2) high-efficiency and high-accuracy 'RoPE' attention based on the tiling optimized FLAT, and (3) highly-fused MLP with fine-grain pipeline scheduling. Our results confirm that FastTPS significantly alleviates memory bottlenecks in the token phase, delivering a 6x speed improvement (compared to none-fusion) on an AMD Ryzen AI 300 series NPU with BF16 precision while sustaining 93% peak memory bandwidth utilization during Phi3-mini-4k-instruct inference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 61 references that are in the map. First posted on arXiv 13 Jul 2026.

Builds on

37 of 61 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  12. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
25 more
  1. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  2. 2024Phi-4 Technical Reportcited by 711
  3. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  6. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  9. 2023Challenges and Applications of Large Language Modelscited by 254
  10. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  11. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  12. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  13. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  14. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  15. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  16. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  17. 2018AI Benchmark: Running Deep Neural Networks on Android Smartphonescited by 75
  18. 2021Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNetcited by 61
  19. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 39
  20. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 39
  21. 2024Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inferencecited by 15
  22. 2024LLMem: Estimating GPU Memory Usage for Fine-Tuning Pre-Trained LLMscited by 12
  23. 2024Is Flash Attention Stable?cited by 10
  24. 2024Understanding Large Language Models in Your Pockets: Performance Study on COTS Mobile Devicescited by 10
  25. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modelingcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV
  2. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  3. 2025SparAMX: Accelerating Compressed LLMs Token Generation on AMX-powered CPUs
  4. 2025MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness
  5. 2026AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall ↗
  6. 2026RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems
  7. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction
  8. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling