# LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training

arXiv: https://arxiv.org/abs/2608.11919 · HTML: https://mlsanity.com/paper/2608.11919/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 35 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://mlsanity.com/paper/2104.07857/) (2021)
- [ZeRO++: Extremely Efficient Collective Communication for Giant Model Training](https://mlsanity.com/paper/2306.10209/) (2023)
- [Efficient Training of Large Language Models on Distributed Infrastructures: A Survey](https://mlsanity.com/paper/2407.20018/) (2024)
- [Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage](https://mlsanity.com/paper/2506.06472/) (2025)
- [10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training](https://mlsanity.com/paper/2511.14124/) (2025)
- [MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU](https://mlsanity.com/paper/2604.05091/) (2026)

## Built on by (most cited first, 1 of 1)

- [HiLoRe: What to Store, Compress, or Recompute for Efficient GRPO Training](https://mlsanity.com/paper/2609.33570/) (2026)

## Abstract (arXiv)

Training large language models on limited hardware is increasingly a scheduling problem across GPU compute, host memory, PCIe transfer, and storage bandwidth. Existing offloading systems reduce GPU residency, and MegaTrain shows that a CPU-master layer-streaming executor can train large models on a single GPU, but fixed checkpointing and placement heuristics still leave communication exposed on the critical path. We propose LazyTrain, an optimization layer over a layer-streaming executor. LazyTrain formulates checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, then executes the solved policy during training. It further couples 8-bit optimizer states with fast gradient clipping as a single Hybrid 8-bit operator: state compression reduces optimizer-state memory, while fast clipping counteracts the additional CPU-side update overhead. Across H800 experiments from Qwen2.5-3B to Qwen3.6-27B, LazyTrain improves sustained TFLOPS over matched baselines runs by approximately 1.24×; RTX 3090 experiments likewise increase the maximum feasible batch size by one at each model scale. In the primary Qwen3.6-27B H800 MetaMathQA run, LazyTrain reaches 219.95 TFLOPS and 1361 tokens/s at batch size 72, peaks at 68.84,GB of GPU memory, and obtains 95.42% exact-match accuracy on the full evaluation split. The source code is available at https://github.com/DataArcTech/LazyTrain.