mlsanityData to 7 Oct 2026

ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving

arXiv 2604.14626 ↗First posted 16 Apr 2026cs.LG cs.AI cs.ARcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Mixture-of-Experts (MoE) models have become the dominant architecture for large-scale language models, yet on-premises serving remains fundamentally memory-bound as batching turns sparse per-token compute into dense memory activation. Memory-centric architectures (PIM, NMP) improve bandwidth but leave compute underutilized under MoE's low arithmetic intensity at high batch sizes. Speculative decoding (SD) trades idle compute for fewer target invocations, yet verification must load experts even for rejected tokens, severely limiting its benefit in MoE especially at low batch sizes. We propose ELMoE-3D, a hybrid-bonding (HB)-based HW-SW co-designed framework that unifies cache-based acceleration and speculative decoding to offer overall speedup across batch sizes. We identify two intrinsic elasticity axes of MoE-expert and bit-and jointly scale them to construct Elastic Self-Speculative Decoding (Elastic-SD), which serves as both an expert cache and a strongly aligned self-draft model accelerated by high HB bandwidth. Our LSB-augmented bit-sliced architecture exploits inherent redundancy in bit-slice representations to natively support bit-nested execution. On our 3D-stacked hardware, ELMoE-3D achieves an average 6.6× speedup and 4.4× energy efficiency gain over naive MoE serving on xPU across batch sizes 1-16, and delivers 2.2× speedup and 1.4× energy efficiency gain over the best-performing prior accelerator baseline.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 37 of its 66 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

37 of 66 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decoding17 Feb
  2. 2025SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference15 Dec
  3. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts18 Nov
  4. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference13 Nov
  5. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference11 Oct
  6. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts17 Jun
  7. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE26 May
  8. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints12 Apr
  9. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference27 Nov
  10. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  11. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching2 Sep
  12. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 25

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  7. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  8. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  9. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  10. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  11. 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 95
  12. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 50
13 more
  1. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 48
  2. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanismcited by 31
  3. 2023Ramulator 2.0: A Modern, Modular, and Extensible DRAM Simulatorcited by 26
  4. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMscited by 21
  5. 2025NestQuant: Nested Lattice Quantization for Matrix Products and LLMscited by 16
  6. 2025Matryoshka Quantizationcited by 11
  7. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  8. 2025LP-Spec: Leveraging LPDDR PIM for Efficient LLM Mobile Speculative Inference with Architecture-Dataflow Co-Optimizationcited by 9
  9. 2025AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMscited by 8
  10. 2024Panacea: Novel DNN Accelerator using Accuracy-Preserving Asymmetric Quantization and Energy-Saving Bit-Slice Sparsitycited by 5
  11. 2025TruncQuant: Truncation-Ready Quantization for DNNs with Flexible Weight Bit Precisioncited by 3
  12. 2025Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decodingcited by 1
  13. 2025SeVeDo: A Heterogeneous Transformer Accelerator for Low-Bit Inference via Hierarchical Group Quantization and SVD-Guided Mixed Precisioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
  2. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
  3. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
  4. 2026BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration
  5. 2025Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding
  6. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  7. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache
  8. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding