# Apple Intelligence Foundation Language Models: Tech Report 2025

arXiv: https://arxiv.org/abs/2507.13575 · HTML: https://mlsanity.com/paper/2507.13575/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 17 Jul 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)
- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints](https://mlsanity.com/paper/2212.05055/) (2022)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [Rope to Nope and Back Again: A New Hybrid Attention Strategy](https://mlsanity.com/paper/2501.18795/) (2025)
- [ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization](https://mlsanity.com/paper/2502.02631/) (2025)
- [Apple Intelligence Foundation Language Models](https://mlsanity.com/paper/2407.21075/) (2024)
- [Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training](https://mlsanity.com/paper/2405.15052/) (2024)
- [AXLearn: Modular, Hardware-Agnostic Large Model Training](https://mlsanity.com/paper/2507.05411/) (2025)

## Built on by (most cited first, 28 of 28)

- [Mid-Training of Large Language Models: A Survey](https://mlsanity.com/paper/2510.06826/) (2025)
- [FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models](https://mlsanity.com/paper/2509.20624/) (2025)
- [Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents](https://mlsanity.com/paper/2509.26539/) (2025)
- [MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer](https://mlsanity.com/paper/2509.16197/) (2025)
- [HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing](https://mlsanity.com/paper/2602.03560/) (2026)
- [Forget to Improve: On-Device LLM-Agent Continual Learning via Budget-Curated Memory](https://mlsanity.com/paper/2606.25115/) (2026)
- [MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers](https://mlsanity.com/paper/2602.00398/) (2026)
- [Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models](https://mlsanity.com/paper/2602.05897/) (2026)
- [Data-Centric Lessons To Improve Speech-Language Pretraining](https://mlsanity.com/paper/2510.20860/) (2025)
- [Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search](https://mlsanity.com/paper/2510.22101/) (2025)
- [Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback](https://mlsanity.com/paper/2603.02232/) (2026)
- [Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference](https://mlsanity.com/paper/2510.13161/) (2025)
- [ZK-APEX: Zero-Knowledge Approximate Personalized Unlearning with Executable Proofs](https://mlsanity.com/paper/2512.09953/) (2025)
- [TOM: A Ternary Read-only Memory Accelerator for LLM-powered Edge Intelligence](https://mlsanity.com/paper/2602.20662/) (2026)
- [FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation](https://mlsanity.com/paper/2603.09046/) (2026)
- [LOCALUT: Harnessing Capacity-Computation Tradeoffs for LUT-Based Inference in DRAM-PIM](https://mlsanity.com/paper/2604.04523/) (2026)
- [Peak-Detector: Explainable Peak Detection via Instruction-Tuned Large Language Models in Physiological Sign](https://mlsanity.com/paper/2605.16452/) (2026)
- [LLM-Augmented Symbolic NLU System for More Reliable Continuous Causal Statement Interpretation](https://mlsanity.com/paper/2510.19988/) (2025)
- [Toward Agentic Environments: GenAI and the Convergence of AI, Sustainability, and Human-Centric Spaces](https://mlsanity.com/paper/2512.15787/) (2025)
- [Enabling Physical AI at the Edge: Hardware-Accelerated Recovery of System Dynamics](https://mlsanity.com/paper/2512.23767/) (2025)
- [WRAVAL -- WRiting Assist eVALuation](https://mlsanity.com/paper/2601.03268/) (2025)
- [Data-driven Clustering and Merging of Adapters for On-device Large Language Models](https://mlsanity.com/paper/2601.17441/) (2026)
- [TriGen: NPU Architecture for End-to-End Acceleration of Large Language Models based on SW-HW Co-Design](https://mlsanity.com/paper/2602.12962/) (2026)
- [PLACID: Privacy-preserving Large language models for Acronym Clinical Inference and Disambiguation](https://mlsanity.com/paper/2603.23678/) (2026)
- [How Vulnerable Are Edge LLMs?](https://mlsanity.com/paper/2603.23822/) (2026)
- [Fortress: A Case Study in Stabilizing Search Recommendations via Temporal Data Augmentation and Feature Pruning](https://mlsanity.com/paper/2605.15299/) (2026)
- [FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation](https://mlsanity.com/paper/2606.23370/) (2026)
- [Fine-Tuning a 3B-Parameter LLM on a Smartphone: Characterizing Sustained Training](https://mlsanity.com/paper/2610.06325/) (2026)

## Abstract (arXiv)

We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services: i a 3B-parameter on-device model optimized for Apple silicon through architectural innovations such as KV-cache sharing and 2-bit quantization-aware training; and ii a scalable server model built on a novel Parallel-Track Mixture-of-Experts PT-MoE transformer that combines track parallelism, mixture-of-experts sparse computation, and interleaved global-local attention to deliver high quality with competitive cost on Apple's Private Cloud Compute platform. Both models are trained on large-scale multilingual and multimodal datasets sourced via responsible web crawling, licensed corpora, and high-quality synthetic data, then further refined with supervised fine-tuning and reinforcement learning on a new asynchronous platform. The resulting models support several additional languages while understanding images and executing tool calls. In public benchmarks and human evaluations, both the server model and the on-device model match or surpass comparably sized open baselines. A new Swift-centric Foundation Models framework exposes guided generation, constrained tool calling, and LoRA adapter fine-tuning, allowing developers to integrate these capabilities with a few lines of code. The latest advancements in Apple Intelligence models are grounded in our Responsible AI approach with safeguards like content filtering and locale-specific evaluation, as well as our commitment to protecting our users' privacy with innovations like Private Cloud Compute.