mlsanityData to 7 Oct 2026

JetMoE: Reaching Llama2 Performance with 0.1M Dollars

arXiv 2404.07413 ↗First posted 11 Apr 2024cs.CL cs.AIcited by 55

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence. This report introduces JetMoE-8B, a new LLM trained with less than 0.1 million, using 1.25T tokens from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the Llama2-13B-Chat model. These results suggest that LLM training can be much more cost-effective than generally thought. JetMoE-8B is based on an efficient Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B to have 8B parameters while only activating 2B for each input token, reducing inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B is highly open and academia-friendly, using only public datasets and training code. All training parameters and data mixtures have been detailed in this report to facilitate future efforts in the development of open foundation models. This transparency aims to encourage collaboration and further advancements in the field of accessible and efficient LLMs. The model weights are publicly available at https://github.com/myshell-ai/JetMoE.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 38 of its 64 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

38 of 64 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models8 Apr
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2023Mistral 7Bcited by 3,065
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
22 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2023StarCoder: may the source be with you!cited by 790
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  13. 2023Instruction Tuning with GPT-4cited by 676
  14. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  15. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  16. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  17. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  18. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  19. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  20. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  21. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  22. 2023Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuningcited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 25; 2026: 9.

55 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  3. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  4. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  5. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  6. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  7. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  8. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  9. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  10. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  11. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  12. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37
38 more
  1. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
  2. 2024Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Schedulercited by 26
  3. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Modelscited by 21
  4. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  5. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 13
  6. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 12
  7. 2024Searching for Efficient Linear Layers over a Continuous Space of Structured Matricescited by 11
  8. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deploymentcited by 10
  9. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 9
  10. 2024GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theorycited by 8
  11. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  12. 2025MergeMoE: Efficient Compression of MoE Models via Expert Output Mergingcited by 8
  13. 2025GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanismcited by 7
  14. 2025Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsitycited by 7
  15. 2025C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization for Test-Time Expert Re-Mixingcited by 7
  16. 2026Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformationcited by 7
  17. 2025MoE2: Optimizing Collaborative Inference for Edge Large Language Modelscited by 6
  18. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Expertscited by 6
  19. 2025Understanding the Mixture-of-Experts with Nadaraya-Watson Kernelcited by 6
  20. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5
  21. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5
  22. 2025Route Experts by Sequence, not by Tokencited by 5
  23. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 5
  24. 2024LaDiMo: Layer-wise Distillation Inspired MoEfiercited by 4
  25. 2025DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptationcited by 4
  26. 2025Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inferencecited by 4
  27. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Modelscited by 4
  28. 2025HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swapcited by 4
  29. 2024Dynamic Activation Pitfalls in LLaMA Models: An Empirical Studycited by 3
  30. 2025Beyond Parameters: Exploring Virtual Logic Depth for Scaling Lawscited by 2
  31. 2025Position: Require Frontier AI Labs To Release Small "Analog" Modelscited by 2
  32. 2026LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacingcited by 2
  33. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimizationcited by 2
  34. 2024More Compute Is What You Needcited by 1
  35. 2025Evaluating Expert Contributions in a MoE LLM for Quiz-Based Taskscited by 0
  36. 2025SHMoAReg: Spark Deformable Image Registration via Spatial Heterogeneous Mixture of Experts and Attention Headscited by 0
  37. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 0
  38. 2025RAST-MoE-RL: A Regime-Aware Spatio-Temporal MoE Framework for Deep Reinforcement Learning in Ride-Hailingcited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE
  2. 2024Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages
  3. 2023Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages
  4. 2025Leveraging the true depth of LLMs
  5. 2024Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment
  6. 20247B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancement
  7. 2024SoupLM: Model Integration in Large Language and Multi-Modal Models
  8. 2023YAYI 2: Multilingual Open-Source Large Language Models