mlsanityData to 14 Oct 2026

Full Parameter Fine-tuning for Large Language Models with Limited Resources

arXiv 2306.09782 ↗First posted 16 Jun 2023cs.CLcited by 134

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) but demand massive GPU resources for training. Lowering the threshold for LLMs training would encourage greater participation from researchers, benefiting both academia and society. While existing approaches have focused on parameter-efficient fine-tuning, which tunes or adds a small number of parameters, few have addressed the challenge of tuning the full parameters of LLMs with limited resources. In this work, we propose a new optimizer, LOw-Memory Optimization (LOMO), which fuses the gradient computation and the parameter update in one step to reduce memory usage. By integrating LOMO with existing memory saving techniques, we reduce memory usage to 10.8% compared to the standard approach (DeepSpeed solution). Consequently, our approach enables the full parameter fine-tuning of a 65B model on a single machine with 8 RTX 3090, each with 24GB memory.Code and data are available at https://github.com/OpenLMLab/LOMO.

Key facts. Cited by 134 papers in the map (data through 14 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

24 of 38 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2017Mixed Precision Trainingcited by 1,156
  10. 2016An overview of gradient descent optimization algorithmscited by 1,033
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  12. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
12 more
  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  2. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  3. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  4. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  5. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  6. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  7. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  8. 2019A Survey of Optimization Methods from a Machine Learning Perspectivecited by 91
  9. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  10. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  11. 2020Training Large Neural Networks with Constant Memory using a New Execution Algorithmcited by 34
  12. 2019Every Local Minimum Value is the Global Minimum Value of Induced Model in Non-convex Machine Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 66; 2025: 34; 2026: 19.

134 papers.

Sort:

By area:

  1. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  2. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  3. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  4. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  5. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  6. 2023Efficient Large Language Models: A Surveycited by 186
  7. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  8. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  9. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  10. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  11. 2024Higher Layers Need More LoRA Expertscited by 102
  12. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Modelscited by 100
38 more
  1. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  2. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  3. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 74
  4. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 68
  5. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 54
  6. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  7. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
  8. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  9. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
  10. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
  11. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  12. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  13. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  14. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 37
  15. 2024Weak-to-Strong Reasoningcited by 33
  16. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 31
  17. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  18. 2023A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrologycited by 26
  19. 2023ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimizationcited by 25
  20. 2024Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applicationscited by 25
  21. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  22. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 22
  23. 2023Sparse is Enough in Fine-tuning Pre-trained Large Language Modelscited by 21
  24. 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuningcited by 21
  25. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  26. 2023Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuningcited by 18
  27. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  28. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 18
  29. 2024PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMscited by 18
  30. 2024Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extractioncited by 18
  31. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 17
  32. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 17
  33. 2024Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuningcited by 16
  34. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 16
  35. 2025Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learningcited by 16
  36. 2024NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networkscited by 14
  37. 2024Pruning as a Domain-specific LLM Extractorcited by 13
  38. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13

50 most cited of 134 All 134 (JSON)