mlsanityData to 14 Oct 2026

Regulating Branch Parallelism in LLM Serving

arXiv 2605.06914 ↗First posted 7 May 2026cs.DC cs.AI cs.CLcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Recent methods expose intra-request parallelism in LLM outputs, allowing independent branches to decode concurrently. Existing serving systems execute these branches eagerly or under fixed caps. We show that both are brittle: eager admission inflates the shared decode step, degrading co-batched requests in serial stages, while conservative fixed caps forgo the throughput that motivated exposing branches in the first place. We call the excess step latency caused by admitted branches the branch externality and show that the safe width depends on batch composition, context lengths, and accumulated slack, all of which change continuously over a workload trace. We introduce TAPER, a per-step admission controller that treats extra branches as opportunistic work, admitted only when the predicted branch externality fits within the batch's current slack budget. Per-step regulation is practical because branch-level scheduling decouples compute from memory: branches share the request's prefix KV, so expanding or contracting width requires no memory reclamation. On Qwen3-32B, TAPER improves goodput by 1.77× over IRP-Off and by 1.48× over IRP-Eager, while maintaining over 95% SLO attainment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 43 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

35 of 43 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production15 May
  2. 2025Hogwild! Inference: Parallel LLM Generation via Concurrent Attention8 Apr
  3. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput22 Aug
  4. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  5. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
  6. 2023Fairness in Serving Large Language Models31 Dec

Other references · 29

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  11. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  12. 2023Universal Self-Consistency for Large Language Model Generationcited by 203
17 more
  1. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 158
  2. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 105
  3. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 61
  4. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58
  5. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodputcited by 37
  6. 2025Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generationcited by 30
  7. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  8. 2025Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decodingcited by 28
  9. 2024APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decodingcited by 24
  10. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Servingcited by 18
  11. 2025ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMscited by 11
  12. 2025Efficient Beam Search for Large Language Models Using Trie-Based Decodingcited by 10
  13. 2025SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Modelscited by 10
  14. 2025ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Modelscited by 8
  15. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  16. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Servingcited by 6
  17. 2025PARALLELPROMPT: Extracting Parallelism from Large Language Model Queriescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale
  2. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  4. 2026RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving
  5. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
  6. 2026Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
  7. 2026SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving
  8. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving