# SlimLM: An Efficient Small Language Model for On-Device Document Assistance

arXiv: https://arxiv.org/abs/2411.09944 · HTML: https://mlsanity.com/paper/2411.09944/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2024.

## Built on by (most cited first, 9 of 9)

- [Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deployment](https://mlsanity.com/paper/2501.03265/) (2025)
- [Toward Autonomous O-RAN: A Multi-Scale Agentic AI Framework for Real-Time Network Control and Management](https://mlsanity.com/paper/2602.14117/) (2026)
- [Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference](https://mlsanity.com/paper/2505.06461/) (2025)
- [Menta: A Small Language Model for On-Device Mental Health Prediction](https://mlsanity.com/paper/2512.02716/) (2025)
- [Kakugo: Distillation of Low-Resource Languages into Small Language Models](https://mlsanity.com/paper/2601.14051/) (2026)
- [DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer](https://mlsanity.com/paper/2606.04694/) (2026)
- [Tiny Enough to Break In: Agentic Remote Access Trojans Powered by Small Language Models](https://mlsanity.com/paper/2608.03009/) (2026)
- [An Empirical Study of On-Device Translation for Real-Time Live-Stream Chat on Mobile Devices](https://mlsanity.com/paper/2601.02641/) (2026)
- [Short-form Text Rewriting with Phi Silica](https://mlsanity.com/paper/2606.00462/) (2026)

## Abstract (arXiv)

While small language models (SLMs) show promises for mobile deployment, their real-world performance and applications on smartphones remains underexplored. We present SlimLM, a series of SLMs optimized for document assistance tasks on mobile devices. Through extensive experiments on a Samsung Galaxy S24, we identify the optimal trade-offs between model size (ranging from 125M to 7B parameters), context length, and inference time for efficient on-device processing. SlimLM is pre-trained on SlimPajama-627B and fine-tuned on DocAssist, our constructed dataset for summarization, question answering and suggestion tasks. Our smallest model demonstrates efficient performance on S24, while larger variants offer enhanced capabilities within mobile constraints. We evaluate SlimLM against existing SLMs, showing comparable or superior performance and offering a benchmark for future research in on-device language models. We also provide an Android application, offering practical insights into SLM deployment. Our findings provide valuable insights and illuminate the capabilities of running advanced language models on high-end smartphones, potentially reducing server costs and enhancing privacy through on-device processing.