mlsanityData to 7 Oct 2026

Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models

arXiv 2604.19767 ↗First posted 27 Mar 2026cs.LG cs.AIcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

We evaluate speculative decoding with EAGLE3 as an inference-time optimization for PayPal's Commerce Agent, powered by a fine-tuned llama3.1-nemotron-nano-8B-v1 model. Building on prior work (NEMO-4-PAYPAL) that reduced latency and cost through domain-specific fine-tuning, we benchmark EAGLE3 via vLLM against NVIDIA NIM on identical 2xH100 hardware across 40 configurations spanning speculative token counts (gamma=3, gamma=5), concurrency levels (1-32), and sampling temperatures (0, 0.5). Key findings: (1) gamma=3 achieves 22-49% throughput improvement and 18-33% latency reduction at zero additional hardware cost; (2) acceptance rates remain stable at approximately 35.5% for gamma=3 across all conditions; (3) gamma=5 yields diminishing returns (approximately 25% acceptance rate); (4) LLM-as-Judge evaluation confirms fully preserved output quality; and (5) speculative decoding on a single H100 matches or exceeds NIM on two H100s, enabling 50% GPU cost reduction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 13 of its 15 references that are in the map. First posted on arXiv 27 Mar 2026.

Builds on

13 of 15 references.

References · 13

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  5. 2024A Survey on LLM-as-a-Judgecited by 1,384
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  9. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  10. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  12. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
1 more
  1. 2025NEMO-4-PAYPAL: Leveraging NVIDIA's Nemo Framework for empowering PayPal's Commerce Agentcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput
  2. 2025Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoff
  3. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  4. 2026Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
  5. 2026FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving
  6. 2025Speculative Decoding: Performance or Illusion?
  7. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving
  8. 2026AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs