mlsanityData to 14 Oct 2026

ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling

arXiv 2608.07974 ↗First posted 8 Aug 2026cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM) fine-tuning at the edge adapts the model to scenario-specific data while preserving privacy. Although existing studies proposed pipeline parallelism to address the limited memory and computing resources of edge devices, they commonly rely on backpropagation (BP) training, which has a fundamental limitation of update locking and could experience severe throughput and memory bottlenecks. In this work, we propose a BP-free algorithm, called ZeroLock, that decouples the model updates into independent chunk updates by local objective construction. It breaks the update locking of BP and hence can improve throughput at the algorithm level and lower memory usage by reducing activation storage. To the best of our knowledge, we provide the first theoretical framework for such local objective construction-based approaches under general model chunk division by mapping local objectives to the global objective. We prove that ZeroLock has a convergence rate of tildeO(1/√T), which differs from BP only by polylogarithmic factors. We design a system for ZeroLock and build real-world prototypes, incorporating techniques such as early forwarding and failure recovery for efficient and robust implementation. Experiments on the prototype show that compared to BP-based baselines, ZeroLock reduces the memory by 26.5% and improves throughput by 4.9%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

21 of 33 references.

References · 21

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  3. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  4. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  5. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  6. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  7. 2020Predictive Coding Approximates Backprop along Arbitrary Computation Graphscited by 90
  8. 2019Deep Learning without Weight Transportcited by 87
  9. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  10. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  11. 2025End-to-End Test-Time Training for Long Contextcited by 62
  12. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
9 more
  1. 2018Decoupled Parallel Backpropagation with Convergence Guaranteecited by 39
  2. 2024Convergence Analysis of Split Federated Learning on Heterogeneous Datacited by 17
  3. 2024Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Paralleliscited by 9
  4. 2024Faster Multi-GPU Training with PPLL: A Pipeline Parallelism Framework Leveraging Local Learningcited by 4
  5. 2026Rethinking Local Learning: A Cheaper and Faster Recipe for LLM Post-Trainingcited by 4
  6. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  7. 2026CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuningcited by 3
  8. 2025CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networkscited by 1
  9. 2026SCPL: Enhancing Neural Network Training Throughput with Decoupled Local Losses and Model Parallelismcited by 1

Built on by

Not cited yet.