mlsanityData to 7 Oct 2026

Tuning of Mixture-of-Experts Mixed-Precision Neural Networks

arXiv 2209.15427 ↗First posted 29 Sep 2022cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Deep learning has become a useful data analysis method, however mainstream adaption in distributed computer software and embedded devices has been low so far. Often, adding deep learning inference in mainstream applications and devices requires new hardware with signal processors suited for convolutional neural networks. This work adds new data types (quantized 16-bit and 8-bit integer, 16-bit floating point) to Caffe in order to save memory and increase inference speed on existing commodity graphics processors with OpenCL, common in everyday devices. Existing models can be executed effortlessly in mixed-precision mode. Additionally, we propose a variation of mixture-of-experts to increase inference speed on AlexNet for image classification. We managed to decrease memory usage up to 3.29x while increasing inference speed up to 3.01x on certain devices. We demonstrate with five simple examples how the presented techniques can easily be applied to different machine learning problems. The whole pipeline, consisting of models, example python scripts and modified Caffe library, is available as Open Source software.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 7 of its 17 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

7 of 17 references.

References · 7

  1. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  4. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  5. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  6. 2014cuDNN: Efficient Primitives for Deep Learningcited by 240
  7. 2015Conditional Computation in Neural Networks for faster modelscited by 166

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  2. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference
  3. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  4. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading
  5. 2022Tutel: Adaptive Mixture-of-Experts at Scale
  6. 2026Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
  7. 2025MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators
  8. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPU