mlsanityData to 7 Oct 2026

ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models

arXiv 2610.01296 ↗First posted 1 Oct 2026cs.AIcited by 0

On the map: Diffusion models → Discrete and text diffusion

also in 2 more
Abstract (arXiv)

Mixture-of-Experts (MoE) Diffusion Language Models (DLMs) offer flexible parallel decoding and increased model capacity, but their large number of expert parameters incurs substantial computation and storage costs. Existing low-rank MoE compression methods largely rely on static factorization and fixed rank allocation, which overlook the distinctive properties of MoE DLMs. Specifically, we identify two properties: cross-mode non-uniform redundancy, where parameter redundancy and sensitivity to rank truncation vary across the input, output, and expert modes, and token-wise utilization variation, where hot and cold tokens exhibit distinct spectral characteristics and expert activation patterns. To address these challenges, we propose ITC-MoE, an Importance-guided Token-aware Compression framework for MoE DLMs. ITC-MoE consists of two complementary components. First, Importance-guided Adaptive Tucker Compression (IATC) incorporates activation and gradient importance into expert weight transformation, jointly factorizes expert weights across multiple modes, and adaptively allocates ranks under a fixed parameter budget. Second, Token-aware Compensation and Routing (TCR) applies lightweight low-rank compensation to compression-sensitive hot tokens and restricts the candidate expert set for cold tokens with concentrated routing patterns. By jointly adapting compression capacity and inference execution to both parameter redundancy and token-wise variation, ITC-MoE substantially reduces the computation and storage costs of MoE DLMs while preserving their generation quality. For example, on SDAR-30B-A3B-Chat-b32, ITC-MoE maintains an accuracy of 96.33% on MultiArith under a 30% compression budget, while achieving up to a 7.22x end-to-end speedup. The code is publicly available at https://github.com/lianjunl13-sudo/ITC-MoE.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 39 of its 47 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

39 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Acceleration9 Feb
  2. 2025Learning Unmasking Policies for Diffusion Language Models9 Dec
  3. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation7 Oct
  4. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models28 Sep
  5. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing8 Aug
  6. 2025Large Language Diffusion Models14 Feb
  7. 2024Simple and Effective Masked Diffusion Language Models11 Jun

Other references · 32

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  5. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  8. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  9. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  10. 2023From Sparse to Soft Mixtures of Expertscited by 245
  11. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  12. 2025Delta Decompression for MoE-based LLMs Compressioncited by 32
20 more
  1. 2025MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidancecited by 32
  2. 2024Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Expertscited by 31
  3. 2024Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budgetcited by 30
  4. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 29
  5. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 26
  6. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 24
  7. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 23
  8. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipelinecited by 20
  9. 2025Dense2MoE: Restructuring Diffusion Transformer to MoE for Efficient Text-to-Image Generationcited by 16
  10. 2025Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Expertscited by 13
  11. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoEcited by 12
  12. 2025Mixture of Lookup Expertscited by 11
  13. 2023Modulate Your Spectrum in Self-Supervised Learningcited by 9
  14. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
  15. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMscited by 9
  16. 2025MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensatorscited by 7
  17. 2026KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Modelscited by 7
  18. 2026Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Densitycited by 6
  19. 2025TuckA: Hierarchical Compact Tensor Experts for Efficient Fine-Tuningcited by 2
  20. 2023Linear shrinkage of sample covariance matrix or matrices under elliptical distributions: a reviewcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More
  2. 2026REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models
  3. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  4. 2025SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
  5. 2025Efficient Training of Diffusion Mixture-of-Experts Models: A Practical Recipe
  6. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
  7. 2026E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models
  8. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Model