mlsanityData to 14 Oct 2026

LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs

arXiv 2512.15306 ↗First posted 17 Dec 2025cs.DC cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We present LLMQ, an end-to-end CUDA/C++ implementation for medium-sized language-model training, e.g. 3B to 32B parameters, on affordable, commodity GPUs. These devices are characterized by low memory availability and slow communication compared to datacentre-grade GPUs. Consequently, we showcase a range of optimizations that target these bottlenecks, including activation checkpointing, offloading, and copy-engine based collectives. LLMQ is able to train or fine-tune a 7B model on a single 16GB mid-range gaming card, or a 32B model on a workstation equipped with 4 RTX 4090s. This is achieved while executing a standard 8-bit training pipeline, without additional algorithmic approximations, and maintaining FLOP utilization of around 50%. The efficiency of LLMQ rivals that of production-scale systems on much more expensive cloud-grade GPUs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 17 Dec 2025.

Builds on

26 of 33 references.

References · 26

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2017Mixed Precision Trainingcited by 1,156
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  10. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  12. 20218-bit Optimizers via Block-wise Quantizationcited by 403
14 more
  1. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  2. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  3. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  4. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  5. 2022FP8 Formats for Deep Learningcited by 196
  6. 2023Microscaling Data Formats for Deep Learningcited by 152
  7. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
  8. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 66
  9. 2024Scaling FP8 training to trillion-token LLMscited by 65
  10. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 33
  11. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 22
  12. 2024MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergencecited by 19
  13. 2025Towards Fully FP8 GEMM LLM Training at Scalecited by 11
  14. 2025GradientStabilizer:Fix the Norm, Not the Gradientcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within 5090cited by 1
  2. 2026One Simple Trick for Improving the Performance of Energy-Limited Local Inference and Trainingcited by 0