mlsanityData to 14 Oct 2026

ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning

arXiv 2605.21177 ↗First posted 20 May 2026cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This work presents textscChunkFT, a memory-efficient fine-tuning framework that reformulates full-parameter fine-tuning around a dynamically activated working set. textscChunkFT enables gradient computation for arbitrary sub-tensors without modifying the network architecture, providing an algorithmic foundation for optimizing arbitrary sub-networks while avoiding standard dense gradient computation. We provide a theoretical convergence analysis of textscChunkFT in the deterministic setting. Empirically, we apply textscChunkFT to fine-tune Llama 3-8B and Llama 3-70B using a single RTX 4090-24GB GPU and 2× H800-80GB GPUs, respectively. Full-parameter fine-tuning of a 7B model with a 1K input length requires only 13.72GB of GPU memory. The results demonstrate the effectiveness of textscChunkFT in memory usage, running time, and optimization quality. Moreover, downstream evaluations on language understanding, mathematical reasoning, and MT-Bench show that textscChunkFT consistently outperforms existing memory-efficient baselines. Notably, textscChunkFT achieves performance comparable to, and in some cases exceeding, full-parameter fine-tuning. Our repository is on https://github.com/misonsky/chunk.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

39 of 51 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
27 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  4. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  9. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  10. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  11. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  12. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  13. 2015Coordinate Descent Algorithmscited by 219
  14. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  15. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  16. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  17. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  18. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 112
  19. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  20. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
  21. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 54
  22. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 31
  23. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  24. 2024HiFT: A Hierarchical Full Parameter Fine-Tuning Strategycited by 12
  25. 2025SSMLoRA: Enhancing Low-Rank Adaptation with State Space Modelcited by 5
  26. 2025Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuningcited by 3
  27. 2026High-Rank Structured Modulation for Parameter-Efficient Fine-Tuningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets
  2. 2024MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter
  3. 2024JORA: JAX Tensor-Parallel LoRA Library for Retrieval Augmented Fine-Tuning
  4. 2025ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory
  5. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Models
  6. 2026Fine-Tuning GPT-5 for GPU Kernel Generation
  7. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuning
  8. 2026Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation