mlsanityData to 14 Oct 2026

BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models

arXiv 2404.02827 ↗First posted 3 Apr 2024cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

This work presents BAdam, an optimization method that leverages the block coordinate descent (BCD) framework with Adam's update rule. BAdam offers a memory efficient approach to the full parameter finetuning of large language models. We conduct a theoretical convergence analysis for BAdam in the deterministic case. Experimentally, we apply BAdam to finetune the Llama 3-8B and Llama 3-70B models using a single RTX3090-24GB GPU and 4 A100-80GB GPUs, respectively. The results confirm BAdam's efficiency in terms of memory usage, running time, and optimization capability. Furthermore, the downstream performance evaluation based on MT-bench and math benchmarks shows that BAdam outperforms existing memory efficient baselines such as LoRA. It also demonstrates that BAdam can achieve comparable or even superior performance compared to Adam. Finally, the ablation study using SGD's update rule illustrates the suitability of BCD for finetuning LLMs. Our code can be easily integrated into any PyTorch-based codebase and is available at https://github.com/Ledzy/BAdam.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 51 of its 76 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

51 of 76 references.

References · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  9. 2023Instruction Tuning with GPT-4cited by 686
  10. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  11. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  12. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  13. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  14. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  15. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  16. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  17. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  18. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  19. 2015Coordinate Descent Algorithmscited by 219
  20. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  21. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  22. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  23. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  24. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 115
  25. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  26. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 112
  27. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  28. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  29. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
  30. 2023Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Rankingcited by 44
  31. 2020jiant: A Software Toolkit for Research on General-Purpose Text Understanding Modelscited by 39
  32. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  33. 2013On the Complexity Analysis of Randomized Block-Coordinate Descent Methodscited by 21
  34. 2017Let's Make Block Coordinate Descent Converge Faster: Faster Greedy Rules, Message-Passing, Active-Set Complexity, and Superlinear Convergencecited by 17
  35. 2024HiFT: A Hierarchical Full Parameter Fine-Tuning Strategycited by 12
  36. 2022Cyclic Block Coordinate Descent With Variance Reduction for Composite Nonconvex Optimizationcited by 9
  37. 2015On the Efficiency of Random Permutation for ADMM and Coordinate Descentcited by 2
  38. 2019Managing Randomization in the Multi-Block Alternating Direction Method of Multipliers for Quadratic Optimizationcited by 2

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 9.

27 papers.

Sort:

By area:

  1. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 37
  2. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 23
  3. 2025Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuningcited by 18
  4. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 16
  5. 2025Cautious Weight Decaycited by 13
  6. 2024SABER: Model-agnostic Backdoor Attack on Chain-of-Thought in Neural Code Generationcited by 10
  7. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 7
  8. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
  9. 2024BTBR: A Bayesian-Theory-Driven Probabilistic-Fuzzy Framework for Implicit Bias Removal in Large Language Modelscited by 4
  10. 2024DynamicRouteGPT: A Real-Time Multi-Vehicle Dynamic Navigation Framework Based on Large Language Modelscited by 3
  11. 2025Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Modelscited by 3
  12. 2025StreamBP: Memory-Efficient Exact Backpropagation for Long Sequence Training of LLMscited by 2
15 more
  1. 2025MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Samplingcited by 2
  2. 2026MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimizationcited by 2
  3. 2025GWT: Scalable Optimizer State Compression for Large Language Model Trainingcited by 1
  4. 2025SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuningcited by 1
  5. 2025MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptationcited by 1
  6. 2025Geometrically Principled Randomized Optimization for Efficient LLM Trainingcited by 1
  7. 2026Schattor: Schatten-family methods for deep learning optimizationcited by 1
  8. 2025Systematic Optimization of Open Source Large Language Models for Mathematical Reasoningcited by 0
  9. 2025AdaFRUGAL: Adaptive Memory-Efficient Training with Dynamic Controlcited by 0
  10. 2026Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Samplingcited by 0
  11. 2026ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuningcited by 0
  12. 2026Towards the Readability of LLM-Generated Codes through Multitask Representation Engineeringcited by 0
  13. 2026Unifying Data, Memory, and Compute Efficiency in LLM training: A Surveycited by 0
  14. 2026Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuningcited by 0
  15. 2026LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignmentcited by 0