mlsanityData to 7 Oct 2026

Staircase Streaming for Low-Latency Multi-Agent Inference

arXiv 2510.05059 ↗First posted 6 Oct 2025cs.AIcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Recent advances in large language models (LLMs) opened up new directions for leveraging the collective expertise of multiple LLMs. These methods, such as Mixture-of-Agents, typically employ additional inference steps to generate intermediate outputs, which are then used to produce the final response. While multi-agent inference can enhance response quality, it can significantly increase the time to first token (TTFT), posing a challenge for latency-sensitive applications and hurting user experience. To address this issue, we propose staircase streaming for low-latency multi-agent inference. Instead of waiting for the complete intermediate outputs from previous steps, we begin generating the final response as soon as we receive partial outputs from these steps. Experimental results demonstrate that staircase streaming reduces TTFT by up to 93% while maintaining response quality.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 31 of its 38 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

31 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Mixture-of-Agents Enhances Large Language Model Capabilities7 Jun
  2. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?28 Feb
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug

Other references · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
16 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  4. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  10. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  12. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  13. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  14. 2023Speculative Decoding with Big Little Decodercited by 163
  15. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  16. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 66

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Speculative Interaction Agents: Building Real-Time Agents with Asynchronous I/O and Speculative Tool Callingcited by 2
  2. 2026Streaming Communication in Multi-Agent Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs
  2. 2026AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?
  3. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration
  4. 2026Token-Level LLM Collaboration via FusionRoute
  5. 2026Scaling Test-Time Compute for Agentic Coding
  6. 2026Rethinking Model Efficiency: Multi-Agent Inference with Large Models
  7. 2025CTTS: Collective Test-Time Scaling
  8. 2026Tiny-Critic RAG: Empowering Agentic Fallback with Parameter-Efficient Small Language Models