# Kimi K3: Open Frontier Intelligence

arXiv: https://arxiv.org/abs/2607.24653 · HTML: https://mlsanity.com/paper/2607.24653/

## Key facts

- Cited by 138 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 85 of its 125 references that are in the map.
- First posted on arXiv 27 Jul 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)
- Mixture-of-experts / [MoE in reinforcement learning and robotics](https://mlsanity.com/moe/moe-in-reinforcement-learning-and-robotics/)

## Builds on (most cited first, 30 of 85)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)

## Built on by (most cited first, 30 of 138)

- [Rethinking On-Policy Distillation of Large Language Models II: One Training Example](https://mlsanity.com/paper/2609.04172/) (2026)
- [Stealing Reasoning Traces from Proprietary LLM APIs](https://mlsanity.com/paper/2608.09867/) (2026)
- [Rethinking Privileged Information in On-Policy Self-Distillation](https://mlsanity.com/paper/2608.18271/) (2026)
- [Extremely Sparse Supervision Incentivizes Reasoning Ability](https://mlsanity.com/paper/2609.04565/) (2026)
- [Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements](https://mlsanity.com/paper/2608.17310/) (2026)
- [Multimodal Duplex Interaction Agent](https://mlsanity.com/paper/2609.08977/) (2026)
- [RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2609.20784/) (2026)
- [EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning](https://mlsanity.com/paper/2608.06197/) (2026)
- [Unlocking Lossless Speedups in LLMs via Discrete Diffusion](https://mlsanity.com/paper/2609.04010/) (2026)
- [DFlow: Enabling Verifier Information Flow in Block Diffusion Speculative Decoding](https://mlsanity.com/paper/2609.06498/) (2026)
- [Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation](https://mlsanity.com/paper/2609.08798/) (2026)
- [Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening](https://mlsanity.com/paper/2609.18708/) (2026)
- [Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations](https://mlsanity.com/paper/2609.19101/) (2026)
- [CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?](https://mlsanity.com/paper/2610.07557/) (2026)
- [ReplaySSM: Cache SSM Inputs, Not State](https://mlsanity.com/paper/2610.13778/) (2026)
- [Agentic Commerce World: An Auditable and Verifiable Environment for Vibe Commerce](https://mlsanity.com/paper/2608.02441/) (2026)
- [Scheduling Mixed RL Rollouts Beyond Prefix Locality](https://mlsanity.com/paper/2608.11152/) (2026)
- [Synthetic Persona Pretraining: Alignment from Token Zero](https://mlsanity.com/paper/2608.13482/) (2026)
- [Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability](https://mlsanity.com/paper/2608.30505/) (2026)
- [Language Models Can Control Their Own Attention](https://mlsanity.com/paper/2609.02737/) (2026)
- [IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier](https://mlsanity.com/paper/2609.10494/) (2026)
- [Magenta: Closing the Loop Between Mathematical Reasoning and Lean Verification](https://mlsanity.com/paper/2609.11319/) (2026)
- [ME-VLM: A Unified VLM for Embodied Cognition and Agent Coordination](https://mlsanity.com/paper/2609.24526/) (2026)
- [AdaHVLA: Adaptive Harnesses for Long-Horizon Vision-Language-Action Execution](https://mlsanity.com/paper/2609.29204/) (2026)
- [Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models](https://mlsanity.com/paper/2609.32069/) (2026)
- [DepthBench: Measuring How Residual Connections Enable More Computational Depth](https://mlsanity.com/paper/2609.32534/) (2026)
- [Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL](https://mlsanity.com/paper/2609.32577/) (2026)
- [How to Loop MoE: Flatten the Experts, Untie the Attention](https://mlsanity.com/paper/2609.35751/) (2026)
- [EvalResearchBench: Can AI Agents Design Their Own Evaluations?](https://mlsanity.com/paper/2610.04184/) (2026)
- [From BERT to Frontier Agents: Eight Years of Language-Model Progress, the Collapse of the Capability-Cost Curve, and the Rise of Task-Targeted Models](https://mlsanity.com/paper/2608.13675/) (2026)

## Abstract (arXiv)

We introduce Kimi K3, a 2.8T parameter Mixture-of-Experts model with 104 billion activated parameters, native vision capabilities, and a 1-million-token context window. Kimi K3 is built on Kimi Delta Attention and Attention Residuals, which improve information flow across sequence length and model depth. Together with Stable LatentMoE, which effectively activates 16 of 896 routed experts per token, and refined training and data recipes, these advances yield an approximately 2.5x improvement in overall scaling efficiency over Kimi K2. Post-training highlights reinforcement learning across general, agentic, and coding domains and multiple reasoning-effort levels, enabling compositional generalization and robust long-horizon execution. At 2.8T scale, Kimi K3 is supported by infrastructure advances in multiple areas: algorithm-system co-design for KDA, perfectly balanced expert-parallel training with efficient memory management, million-token agentic RL with persistent rollout and sandbox states, and deployment innovations. Extensive evaluations show that Kimi K3 achieves frontier-level performance across long-horizon coding, agentic, knowledge, reasoning, and vision tasks. While its overall performance still trails the most powerful proprietary models, namely Claude Fable 5 and GPT-5.6 Sol, Kimi K3 consistently outperforms other open and proprietary models evaluated in our suite. We release the full Kimi K3 model weights to facilitate future research and accelerate the broader deployment and adoption of frontier intelligence.