mlsanityData to 7 Oct 2026

Intra-Prompt Parallel Decoding for Common-Context Question Answering

arXiv 2609.05707 ↗First posted 4 Sep 2026cs.CLcited by 1

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

In common-context question answering (CCQA) tasks, multiple input questions share a common context to base their answers from. However, Large Language Models typically generate each answer using an independent prompt. While existing batching and caching techniques help improve parallelism and reduce repeated computations, the separation of questions across prompts limits the achievable speedup, as modern GPUs are underutilized due to a memory bottleneck during attention. We present Intra-Prompt Parallel Decoding (IPPD), a novel inference method that answers multiple common-context questions in parallel within a single prompt. IPPD directly addresses the bottleneck by efficiently sharing both memory and computation during the attention process, as the next token for every question is decoded in a single inference step. IPPD uses virtual position IDs and attention mask manipulation to generate the same output as standard prompting without requiring fine-tuning or any changes to the LLM architecture. Since all parallelism occurs within a prompt, IPPD is fully compatible with batched inference, even when each prompt features a different context. Our experiments show that IPPD delivers up to 7X the effective throughput as standard decoding without quality degradation, and outperforms prefix caching with PagedAttention in most settings.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 26 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

19 of 26 references.

References · 19

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  5. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  7. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  8. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  9. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  10. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 103
  11. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  12. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
7 more
  1. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  2. 2025A Survey of Large Language Model Agents for Question Answeringcited by 46
  3. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 32
  4. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  5. 2024APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decodingcited by 23
  6. 2023ExtractGPT: Exploring the Potential of Large Language Models for Product Attribute Value Extractioncited by 20
  7. 2024DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inferencecited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Scaling E-Commerce Attribute Extraction with Parallel Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs
  2. 2025PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries
  3. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference
  4. 2024Plato: Plan to Efficiently Decode for Large Language Model Inference
  5. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference
  6. 2022Parallel Context Windows for Large Language Models
  7. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
  8. 2025Accelerate Parallelizable Reasoning via Parallel Decoding within One Sequence