mlsanityData to 14 Oct 2026

Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence

arXiv 2607.20981 ↗First posted 23 Jul 2026cs.AIcited by 0

On the map: Mixture-of-experts → Multimodal MoE models

also in 1 more
Abstract (arXiv)

Efficient multimodal inference is increasingly constrained not only by model quality or FLOP count, but also by the cost of preserving, moving, routing, caching, and quantizing multimodal representations under latency, memory, and energy constraints. This paper reviews recent advances in efficient vision-language and multimodal large language models, covering visual token compression, video token management, KV-cache optimization, Mixture-of-Experts (MoE) routing, low-bit quantization, edge deployment, and hardware-aware benchmarking. We argue that these techniques cannot be treated as independent optimizations. Visual token compression alters downstream feature distributions and MoE routing decisions, routing behavior affects expert utilization and quantization sensitivity, quantized router logits influence expert assignment, KV-cache policies determine retained multimodal evidence, and hardware constraints often transform computational savings into memory and communication bottlenecks. We organize the literature around these interactions and identify key design trade-offs, including accuracy versus token budget, static versus adaptive compression, sparse routing efficiency versus expert collapse, and low-bit inference versus modality-specific degradation. Finally, we introduce Temporal Routing Consistency as a diagnostic for video MoE models and highlight open research directions in routing-aware compression, cross-modal cache management, hardware-aware co-design, and unified benchmarking for multimodal edge intelligence.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 87 of its 97 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

87 of 97 references.

References · 87

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  9. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  11. 2022Token Merging: Your ViT But Fastercited by 1,021
  12. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
38 more
  1. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  2. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  3. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 80
  4. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  5. 2024Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Modelcited by 68
  6. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  7. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
  8. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  9. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  10. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inferencecited by 42
  11. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  12. 2024Backdooring Vision-Language Models with Out-Of-Distribution Datacited by 33
  13. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 32
  14. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 30
  15. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Modelscited by 27
  16. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameterscited by 25
  17. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  18. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Servingcited by 20
  19. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Schedulingcited by 19
  20. 2026VisionTrim: Unified Vision Token Compression for Training-Free MLLM Accelerationcited by 18
  21. 2024HoME: Hierarchy of Multi-Gate Experts for Multi-Task Learning at Kuaishoucited by 16
  22. 2024Video Representation Learning with Joint-Embedding Predictive Architecturescited by 15
  23. 2025Mixture of Lookup Expertscited by 12
  24. 2025VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Modelscited by 12
  25. 2025MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inferencecited by 12
  26. 2024HERMES: temporal-coHERent long-forM understanding with Episodes and Semanticscited by 11
  27. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  28. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  29. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  30. 2024Representation Collapsing Problems in Vector Quantizationcited by 10
  31. 2025Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshapecited by 10
  32. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimizationcited by 10
  33. 2025StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understandingcited by 10
  34. 2026Where Do the Joules Go? Diagnosing Inference Energy Consumptioncited by 10
  35. 2024Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Trainingcited by 8
  36. 2024PalmBench: A Comprehensive Benchmark of Compressed Large Language Models on Mobile Platformscited by 8
  37. 2025SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Modelscited by 8
  38. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8

50 most cited of 87 All 87 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Token Sequence Compression for Efficient Multimodal Computing
  2. 2025Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression
  3. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models
  4. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  5. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  6. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI
  7. 2026DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference
  8. 2025AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs