mlsanityData to 7 Oct 2026

Stable and low-precision training for large-scale vision-language models

arXiv 2304.13013 ↗First posted 25 Apr 2023cs.LG cs.CVcited by 74

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

We introduce new methods for 1) accelerating and 2) stabilizing training for large language-vision models. 1) For acceleration, we introduce SwitchBack, a linear layer for int8 quantized training which provides a speed-up of 13-25% while matching the performance of bfloat16 training within 0.1 percentage points for the 1B parameter CLIP ViT-Huge -- the largest int8 training to date. Our main focus is int8 as GPU support for float8 is rare, though we also analyze float8 training through simulation. While SwitchBack proves effective for float8, we show that standard techniques are also successful if the network is trained and initialized so that large feature magnitudes are discouraged, which we accomplish via layer-scale initialized with zeros. 2) For stability, we analyze loss spikes and find they consistently occur 1-8 iterations after the squared gradients become under-estimated by their AdamW second moment estimator. As a result, we recommend an AdamW-Adafactor hybrid which avoids loss spikes when training a CLIP ViT-Huge model and outperforms gradient clipping at the scales we test.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 81 references that are in the map. First posted on arXiv 25 Apr 2023.

Builds on

66 of 81 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  13. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  14. 2017Mixed Precision Trainingcited by 1,135
  15. 2021Scaling Vision Transformerscited by 993
  16. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  17. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  18. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  19. 2019On the Convergence of Adam and Beyondcited by 834
  20. 2021Going deeper with Image Transformerscited by 759
  21. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  22. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  23. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  24. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  25. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  26. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  27. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  28. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  29. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  30. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  31. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  32. 2016Trained Ternary Quantizationcited by 203
  33. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  34. 2022FP8 Formats for Deep Learningcited by 187
  35. 2020Binary Neural Networks: A Surveycited by 184
  36. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  37. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  38. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 24; 2025: 29; 2026: 12.

74 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  3. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  4. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  5. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  6. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 144
  7. 2024Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detectioncited by 92
  8. 2024Scaling Laws for Precisioncited by 92
  9. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  10. 2025REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjectscited by 70
  11. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  12. 2023Intriguing Properties of Quantization at Scalecited by 48
38 more
  1. 2025Exploring Intrinsic Normal Prototypes within a Single Image for Universal Anomaly Detectioncited by 47
  2. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  3. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  4. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 36
  5. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  6. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  7. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 30
  8. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  9. 2025Clinical ModernBERT: An efficient and long context encoder for biomedical textcited by 26
  10. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 21
  11. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 20
  12. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  13. 2025HALO: Hadamard-Assisted Lower-Precision Optimization for LLMscited by 14
  14. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  15. 2024Methods of improving LLM training stabilitycited by 13
  16. 2024COMQ: A Backpropagation-Free Algorithm for Post-Training Quantizationcited by 12
  17. 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
  18. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  19. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  20. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Modelscited by 10
  21. 2024Collage: Light-Weight Low-Precision Strategy for LLM Trainingcited by 9
  22. 2023How to guess a gradientcited by 7
  23. 2024FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resourcescited by 7
  24. 2024FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Trainingcited by 7
  25. 2025INP-Former++: Advancing Universal Anomaly Detection via Intrinsic Normal Prototypes and Residual Learningcited by 7
  26. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 7
  27. 2025GradientStabilizer:Fix the Norm, Not the Gradientcited by 5
  28. 2023How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Studycited by 4
  29. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 4
  30. 2026Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikescited by 4
  31. 2024Robust Training of Neural Networks at Arbitrary Precision and Sparsitycited by 3
  32. 2025AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clippingcited by 3
  33. 2025RoBiS: Robust Binary Segmentation for High-Resolution Industrial Imagescited by 3
  34. 2025DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Modelscited by 3
  35. 2025A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Modelscited by 3
  36. 2024Efficient Training with Denoised Neural Weightscited by 2
  37. 2025Knowing When to Quit: Probabilistic Early Exits for Speech Separationcited by 2
  38. 2026Exploring Silent Data Corruption as a Reliability Challenge in LLM Trainingcited by 2

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models
  2. 2023Effective End-to-End Vision Language Pretraining with Semantic Visual Loss
  3. 2025Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Study
  4. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment
  5. 2024Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation
  6. 2025LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models
  7. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  8. 2026Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients