# Ministral 3

arXiv: https://arxiv.org/abs/2601.08584 · HTML: https://mlsanity.com/paper/2601.08584/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 39 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Direct Language Model Alignment from Online AI Feedback](https://mlsanity.com/paper/2402.04792/) (2024)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [Are We Done with MMLU?](https://mlsanity.com/paper/2406.04127/) (2024)
- [Pixtral 12B](https://mlsanity.com/paper/2410.07073/) (2024)
- [Compact Language Models via Pruning and Knowledge Distillation](https://mlsanity.com/paper/2407.14679/) (2024)
- [LLM Pruning and Distillation in Practice: The Minitron Approach](https://mlsanity.com/paper/2408.11796/) (2024)

## Built on by (most cited first, 30 of 114)

- [The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks](https://mlsanity.com/paper/2603.05498/) (2026)
- [Capabilities Ain't All You Need: Measuring Propensities in AI](https://mlsanity.com/paper/2602.18182/) (2026)
- [EndoCoT: Scaling Endogenous Chain-of-Thought Reasoning in Diffusion Models](https://mlsanity.com/paper/2603.12252/) (2026)
- [TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2603.12529/) (2026)
- [Causal Tracing of Audio-Text Fusion in Large Audio Language Models](https://mlsanity.com/paper/2603.13768/) (2026)
- [Extremely Sparse Supervision Incentivizes Reasoning Ability](https://mlsanity.com/paper/2609.04565/) (2026)
- [Don't Trust Stubborn Neighbors: A Security Framework for Agentic Networks](https://mlsanity.com/paper/2603.15809/) (2026)
- [CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing](https://mlsanity.com/paper/2605.02910/) (2026)
- [Skill Neologisms: Towards Skill-based Continual Learning](https://mlsanity.com/paper/2605.04970/) (2026)
- [ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention](https://mlsanity.com/paper/2605.23081/) (2026)
- [Enhancing Table Reasoning with Deterministic Table-State Rewards](https://mlsanity.com/paper/2601.22530/) (2026)
- [MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI](https://mlsanity.com/paper/2603.19993/) (2026)
- [Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling](https://mlsanity.com/paper/2604.25578/) (2026)
- [Reliable Chain-of-Thought via Prefix Consistency](https://mlsanity.com/paper/2605.07654/) (2026)
- [DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding](https://mlsanity.com/paper/2605.08888/) (2026)
- [To Call or Not to Call: Diagnosing Intrinsic Over-Calling Bias in LLM Agents](https://mlsanity.com/paper/2605.18882/) (2026)
- [UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loop](https://mlsanity.com/paper/2601.21000/) (2026)
- [Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind](https://mlsanity.com/paper/2602.13832/) (2026)
- [Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution](https://mlsanity.com/paper/2602.16154/) (2026)
- [Classroom Final Exam: An Instructor-Tested Reasoning Benchmark](https://mlsanity.com/paper/2602.19517/) (2026)
- [FDARxBench: Benchmarking Regulatory and Clinical Reasoning on FDA Generic Drug Assessment](https://mlsanity.com/paper/2603.19539/) (2026)
- [Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks](https://mlsanity.com/paper/2605.01417/) (2026)
- [On Adaptivity in Zeroth-Order Optimization](https://mlsanity.com/paper/2605.03869/) (2026)
- [Dr-CiK: A Benchmark testing Deep Research for Context-Aided Forecasting](https://mlsanity.com/paper/2605.27904/) (2026)
- [Deep Research as Rubric for Reinforcement Learning](https://mlsanity.com/paper/2606.01091/) (2026)
- [Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation](https://mlsanity.com/paper/2606.05988/) (2026)
- [Language Models Represent and Transform Concepts with Shared Geometry](https://mlsanity.com/paper/2607.04525/) (2026)
- [Evidence Integration in Large Language Models](https://mlsanity.com/paper/2609.04290/) (2026)
- [CORE: Collaborative Reasoning via Cross Teaching](https://mlsanity.com/paper/2601.21600/) (2026)
- [Beyond Holistic Scores: Automatic Trait-Based Quality Scoring of Argumentative Essays](https://mlsanity.com/paper/2602.04604/) (2026)

## Abstract (arXiv)

We introduce the Ministral 3 series, a family of parameter-efficient dense language models designed for compute and memory constrained applications, available in three model sizes: 3B, 8B, and 14B parameters. For each model size, we release three variants: a pretrained base model for general-purpose use, an instruction finetuned, and a reasoning model for complex problem-solving. In addition, we present our recipe to derive the Ministral 3 models through Cascade Distillation, an iterative pruning and continued training with distillation technique. Each model comes with image understanding capabilities, all under the Apache 2.0 license.