mlsanityData to 7 Oct 2026

Mixtral of Experts

arXiv 2401.04088 ↗First posted 8 Jan 2024cs.LG cs.CLcited by 1,785

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.

Key facts. Cited by 1,785 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 32 of its 35 references that are in the map. First posted on arXiv 8 Jan 2024.

Builds on

32 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2023Mistral 7Bcited by 3,065
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
18 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  8. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  10. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  11. 2023Llemma: An Open Language Model For Mathematicscited by 385
  12. 2018QuAC : Question Answering in Contextcited by 379
  13. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  14. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  15. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  16. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  17. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  18. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 764; 2025: 597; 2026: 424.

1,785 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  5. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  6. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  9. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  11. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  12. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
38 more
  1. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  2. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  3. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  4. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  5. 2024The Platonic Representation Hypothesiscited by 362
  6. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  7. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  8. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  9. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  10. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  11. 2024Massive Activations in Large Language Modelscited by 277
  12. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  13. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  14. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  15. 2024Generative Representational Instruction Tuningcited by 227
  16. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  17. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  18. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  19. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  20. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  21. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  22. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  23. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  24. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  25. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
  26. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  27. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  28. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  29. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  30. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  31. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  32. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  33. 2025Qwen2.5-1M Technical Reportcited by 158
  34. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  35. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  36. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  37. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  38. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143

50 most cited of 1,785 All 1,785 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Aurora:Activating Chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning
  2. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
  3. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
  4. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  5. 2025Rethinking Sparse Mixture of Experts from a Unified Perspective
  6. 2024HMoE: Heterogeneous Mixture of Experts for Language Modeling
  7. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
  8. 2025Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures