mlsanityData to 7 Oct 2026

Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs

arXiv 2607.04371 ↗First posted 5 Jul 2026cs.AIcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment. We designed the model to maximize server throughput under high user throughput constraints. In interactive serving workloads on a single 8xB200 node, Puzzle-75B-A9B achieves approximately 2x higher server throughput than Nemotron-3-Super at matched user throughput constraints. In ultra-long-context deployment on a single H100 GPU, the compressed model increases 1M-token concurrency from 1 request to 8 requests. Puzzle-75B-A9B is constructed using a multi-stage pipeline that combines the Iterative Puzzle compression framework with knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head. The compression process jointly optimizes heterogeneous MoE pruning, active parameter budget, and Mamba pruning to improve inference efficiency while preserving model quality. We evaluate Puzzle-75B-A9B on a broad suite of reasoning, coding, multilingual, long-context, and agentic benchmarks. Despite substantial compression, the model retains strong downstream accuracy relative to the parent model across a wide range of tasks. These results demonstrate that large hybrid MoE models can be substantially optimized for deployment efficiency while maintaining strong downstream capability. Our model is publicly available on Hugging Face.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 11 of its 15 references that are in the map. First posted on arXiv 5 Jul 2026.

Builds on

11 of 15 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control8 May
  2. 2026Extending Puzzle for Mixture-of-Experts Reasoning Models with Application to GPT-OSS Acceleration12 Feb
  3. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts26 Jan

Other references · 8

  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  4. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  5. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  6. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 54
  7. 2024Puzzle: Distillation-Based NAS for Inference-Optimized LLMscited by 17
  8. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decodingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Low-Discrepancy Dither for Quantized Recurrent State Cachescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TopoCompress: Topology Aware Token Compression Algorithm for Distributed Edge MoE Inference
  2. 2025Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
  3. 2026Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment
  4. 2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
  5. 2026Joint Encoding of KV-Cache Blocks for Scalable LLM Serving
  6. 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance
  7. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  8. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling