# PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training

arXiv: https://arxiv.org/abs/2411.05046 · HTML: https://mlsanity.com/paper/2411.05046/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 63 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Builds on (most cited first, 20 of 20)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://mlsanity.com/paper/2309.05653/) (2023)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023)
- [APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets](https://mlsanity.com/paper/2406.18518/) (2024)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [LLM in a flash: Efficient Large Language Model Inference with Limited Memory](https://mlsanity.com/paper/2312.11514/) (2023)
- [Small Language Models: Survey, Measurements, and Insights](https://mlsanity.com/paper/2409.15790/) (2024)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Fast On-device LLM Inference with NPUs](https://mlsanity.com/paper/2407.05858/) (2024)

## Built on by (most cited first, 7 of 7)

- [Minions: Cost-efficient Collaboration Between On-device and Cloud Language Models](https://mlsanity.com/paper/2502.15964/) (2025)
- [DroidCall: A Dataset for LLM-powered Android Intent Invocation](https://mlsanity.com/paper/2412.00402/) (2024)
- [Every Software as an Agent: Blueprint and Case Study](https://mlsanity.com/paper/2502.04747/) (2025)
- [NOEM3A: a Neuro-symbolic Ontology-Enhanced Method for Multi-intent understanding in Mobile Agents](https://mlsanity.com/paper/2511.19780/) (2025)
- [PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devices](https://mlsanity.com/paper/2601.11553/) (2025)
- [Performance of Small Language Model Pretraining on FABRIC: An Empirical Study](https://mlsanity.com/paper/2602.02632/) (2026)
- [TRACE-BN: Transferring Bangla-English Tutoring Behavior to a Sub-1B Offline Language Model](https://mlsanity.com/paper/2608.15223/) (2026)

## Abstract (arXiv)

The interest in developing small language models (SLM) for on-device deployment is fast growing. However, the existing SLM design hardly considers the device hardware characteristics. Instead, this work presents a simple yet effective principle for SLM design: architecture searching for (near-)optimal runtime efficiency before pre-training. Guided by this principle, we develop PhoneLM SLM family (currently with 0.5B and 1.5B versions), that acheive the state-of-the-art capability-efficiency tradeoff among those with similar parameter size. We fully open-source the code, weights, and training datasets of PhoneLM for reproducibility and transparency, including both base and instructed versions. We also release a finetuned version of PhoneLM capable of accurate Android Intent invocation, and an end-to-end Android demo. All materials are available at https://github.com/UbiquitousLearning/PhoneLM.