mlsanityData to 7 Oct 2026

On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters

arXiv 2606.02437 ↗First posted 1 Jun 2026cs.LG cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) is usually treated as a cheaper alternative to full fine-tuning. We study a broader role: small trainable adapters as persistent local state on top of strong shared foundation models. In this framing, the base model provides shared competence while adapters carry instance-specific behavior such as preferences, skills, tool habits, and memory-like updates. We organize the problem around three scaling axes: Scale Up, where stronger shared priors make small local updates more useful; Scale Down, where we study how small adapters can be while remaining reliable; and Scale Out, where many persistent adapted instances coexist. MinT provides one infrastructure example for managing adapter identity, revision, provenance, evaluation, and serving residency. Together, the results suggest that PEFT can be a compact substrate for persistent personal models rather than only a budget substitute for full fine-tuning.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

30 of 52 references.

References · 30

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  11. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  12. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
18 more
  1. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  2. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  3. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  4. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  5. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  6. 2026Self-Distillation Enables Continual Learningcited by 212
  7. 2022Learning by Distilling Contextcited by 128
  8. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  9. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 52
  10. 2024OLoRA: Orthonormal Low-Rank Adaptation of Large Language Modelscited by 39
  11. 2025Evaluating Parameter Efficient Methods for RLVRcited by 12
  12. 2026δ-mem: Efficient Online Memory for Large Language Modelscited by 12
  13. 2026Understanding LoRA as Knowledge Memory: An Empirical Analysiscited by 11
  14. 2024RankAdaptor: Hierarchical Rank Allocation for Efficient Fine-Tuning Pruned LLMs via Performance Modelcited by 7
  15. 2025Enhancing LLM-Based Social Bot via an Adversarial Learning Frameworkcited by 7
  16. 2025Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchiescited by 5
  17. 2026AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuningcited by 5
  18. 2025Large Language Model Compression with Global Rank and Sparsity Optimizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Paying to Know: Micro-Transaction Markets for Verified Product Information in Agentic E-Commercecited by 0
  2. 2026Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVRcited by 0
  3. 2026Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRAcited by 0