# On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters

arXiv: https://arxiv.org/abs/2606.02437 · HTML: https://mlsanity.com/paper/2606.02437/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Builds on (most cited first, 30 of 30)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://mlsanity.com/paper/2507.05257/) (2025)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [Learning by Distilling Context](https://mlsanity.com/paper/2209.15189/) (2022)
- [LoRA vs Full Fine-tuning: An Illusion of Equivalence](https://mlsanity.com/paper/2410.21228/) (2024)
- [The Path Not Taken: RLVR Provably Learns Off the Principals](https://mlsanity.com/paper/2511.08567/) (2025)
- [OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2406.01775/) (2024)
- [δ-mem: Efficient Online Memory for Large Language Models](https://mlsanity.com/paper/2605.12357/) (2026)
- [Evaluating Parameter Efficient Methods for RLVR](https://mlsanity.com/paper/2512.23165/) (2025)
- [Understanding LoRA as Knowledge Memory: An Empirical Analysis](https://mlsanity.com/paper/2603.01097/) (2026)
- [RankAdaptor: Hierarchical Rank Allocation for Efficient Fine-Tuning Pruned LLMs via Performance Model](https://mlsanity.com/paper/2406.15734/) (2024)
- [Enhancing LLM-Based Social Bot via an Adversarial Learning Framework](https://mlsanity.com/paper/2508.17711/) (2025)
- [Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchies](https://mlsanity.com/paper/2505.03802/) (2025)
- [AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuning](https://mlsanity.com/paper/2602.22268/) (2026)
- [Large Language Model Compression with Global Rank and Sparsity Optimization](https://mlsanity.com/paper/2505.03801/) (2025)

## Built on by (most cited first, 3 of 3)

- [Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA](https://mlsanity.com/paper/2608.09819/) (2026)
- [Paying to Know: Micro-Transaction Markets for Verified Product Information in Agentic E-Commerce](https://mlsanity.com/paper/2606.24783/) (2026)
- [Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR](https://mlsanity.com/paper/2606.31813/) (2026)

## Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) is usually treated as a cheaper alternative to full fine-tuning. We study a broader role: small trainable adapters as persistent local state on top of strong shared foundation models. In this framing, the base model provides shared competence while adapters carry instance-specific behavior such as preferences, skills, tool habits, and memory-like updates. We organize the problem around three scaling axes: Scale Up, where stronger shared priors make small local updates more useful; Scale Down, where we study how small adapters can be while remaining reliable; and Scale Out, where many persistent adapted instances coexist. MinT provides one infrastructure example for managing adapter identity, revision, provenance, evaluation, and serving residency. Together, the results suggest that PEFT can be a compact substrate for persistent personal models rather than only a budget substitute for full fine-tuning.