mlsanityData to 14 Oct 2026

APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding

arXiv 2401.06761 ↗First posted 12 Jan 2024cs.CLcited by 24

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

The massive adoption of large language models (LLMs) demands efficient deployment strategies. However, the auto-regressive decoding process, which is fundamental to how most LLMs generate text, poses challenges to achieve efficient serving. In this work, we introduce a parallel auto-regressive generation method. By instruct-tuning on general domain data that contains hierarchical structures, we enable LLMs to independently plan their generation process and perform auto-parallel auto-regressive (APAR) generation, significantly reducing the number of generation steps. APAR alone can achieve up to 2x speed-up, and when combined with speculative decoding, the speed-up can reach up to 4x. In addition, APAR reduces the key-value cache consumption and attention computation during generation. This leads to a throughput increase of 20-70% and a latency reduce of 20-35% in high-throughput scenarios, compared to state-of-the-art serving frameworks.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 24 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

12 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2020Reformer: The Efficient Transformercited by 1,668
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  10. 2017Non-Autoregressive Neural Machine Translationcited by 463
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 7.

24 papers.

Sort:

By area:

  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  4. 2025Hogwild! Inference: Parallel LLM Generation via Concurrent Attentioncited by 32
  5. 2025Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decodingcited by 28
  6. 2024DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inferencecited by 19
  7. 2024Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matchingcited by 18
  8. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inferencecited by 16
  9. 2024An LLM-Tool Compiler for Fused Parallel Function Callingcited by 15
  10. 2025ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMscited by 11
  11. 2025A Survey on Parallel Reasoningcited by 11
  12. 2025Planned Diffusioncited by 9
12 more
  1. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  2. 2025Sharing State Between Prompts and Programscited by 2
  3. 2025Pipelined Decoder for Efficient Context-Aware Text Generationcited by 1
  4. 2025Confidence-Modulated Speculative Decoding for Large Language Modelscited by 1
  5. 2026Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoningcited by 1
  6. 2026Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extractioncited by 1
  7. 2026Message Passing Enables Efficient Reasoningcited by 1
  8. 2026PaDoc: Layout-Grounded Parallel Decoding for Document Parsingcited by 1
  9. 2026Intra-Prompt Parallel Decoding for Common-Context Question Answeringcited by 1
  10. 2025Parallel Decoder Transformer: Planner-Conditioned Latent Coordination for Model-Intrinsic Parallel Generationcited by 0
  11. 2026Regulating Branch Parallelism in LLM Servingcited by 0
  12. 2026SchemaWeaver: Accelerating On-Device Mobile Agentic AI via Schema-Guided Parallel LLM Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decoding
  2. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
  3. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  4. 2024Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding
  5. 2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models
  6. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  7. 2025AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism
  8. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing