mlsanityData to 7 Oct 2026

Apple Intelligence Foundation Language Models

arXiv 2407.21075 ↗First posted 29 Jul 2024cs.AI cs.CL cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large server-based language model designed for Private Cloud Compute. These models are designed to perform a wide range of tasks efficiently, accurately, and responsibly. This report describes the model architecture, the data used to train the model, the training process, how the models are optimized for inference, and the evaluation results. We highlight our focus on Responsible AI and how the principles are applied throughout the model development.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 38 of its 60 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

38 of 60 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
26 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,425
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  12. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  13. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  14. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  15. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  16. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  17. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  18. 2016Using the Output Embedding to Improve Language Modelscited by 266
  19. 2019Structured Pruning of Large Language Modelscited by 201
  20. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  21. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  22. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  23. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  24. 2021Improved Regret Bound and Experience Replay in Regularized Policy Iterationcited by 18
  25. 2024Talaria: Interactively Optimizing Machine Learning Models for Efficient Inferencecited by 6
  26. 2024Large Language Model-guided Document Selectioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 12; 2026: 7.

22 papers.

Sort:

By area:

  1. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  2. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  3. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 108
  4. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  5. 2025Distillation Scaling Lawscited by 59
  6. 2025Reinforcement Learning from Human Feedbackcited by 53
  7. 2025Apple Intelligence Foundation Language Models: Tech Report 2025cited by 25
  8. 2025Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agentscited by 24
  9. 2026GraphPlanner: Graph Memory-Augmented Agentic Routing for Multi-Agent LLMscited by 10
  10. 2026SpecSteer: Synergizing Local Context and Global Reasoning for Efficient Personalized Generationcited by 6
  11. 2025A Novel Hat-Shaped Device-Cloud Collaborative Inference Framework for Large Language Modelscited by 5
  12. 2025Do Large Language Models Reason Causally Like Us? Even Better?cited by 4
10 more
  1. 2025Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Modelscited by 3
  2. 2024Generative AI Toolkit -- a framework for increasing the quality of LLM-based applications over their whole life cyclecited by 2
  3. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  4. 2026Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotationcited by 2
  5. 2025CG-TTRL: Context-Guided Test-Time Reinforcement Learning for On-Device Large Language Modelscited by 1
  6. 2025FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Modelscited by 1
  7. 2026Continually self-improving AIcited by 0
  8. 2026BiSCo-LLM: Lookup-Free Binary Spherical Coding for Extreme Low-Bit Large Language Model Compressioncited by 0
  9. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devicescited by 0
  10. 2026ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMscited by 0