mlsanityData to 7 Oct 2026

RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving

arXiv 2606.22840 ↗First posted 22 Jun 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

We present RLM-Cascade, a proxy-layer system that applies speculative decoding at the response level to reduce LLM API costs without requiring model architecture access or a shared vocabulary. A fast, inexpensive draft model generates a candidate response; a capable verify model accepts, enhances, or is bypassed entirely depending on a lightweight complexity router. On a real-world agentic coding workload (Claude Code), RLM-Cascade achieves a draft-use rate of 88.8% across 125 production requests, reducing API cost by 45.8% relative to a direct Opus baseline. Counter-intuitively, the proxy also reduces end-to-end latency: median response time is 2,026 ms versus 3,698 ms for Native Opus -- a 1.83X speedup at p50 -- because the SKIPPED path (DeepSeek only, no Opus call) dominates the workload distribution. Quality matches or exceeds the Opus baseline: 100% pass rate on a 20-task Code/Math/Instruct benchmark versus 95% for Native Opus. We further describe a rule-based complexity router that selects the SKIPPED path for simple agentic turns and a hybrid tool-call strategy that bypasses the speculative pipeline for schema-critical tool-selection turns. RLM-Cascade is deployed in production as an enterprise AI infrastructure component and published as open source with a live metrics dashboard and Prometheus endpoint.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 10 of its 13 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

10 of 13 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  3. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  4. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 6

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  6. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  2. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding
  3. 2025AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
  4. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  5. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving
  6. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration
  7. 2026Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
  8. 2025SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models