mlsanityData to 14 Oct 2026

lo-fi: distributed fine-tuning without communication

arXiv 2210.11948 ↗First posted 19 Oct 2022cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

When fine-tuning large neural networks, it is common to use multiple nodes and to communicate gradients at each optimization step. By contrast, we investigate completely local fine-tuning, which we refer to as lo-fi. During lo-fi, each node is fine-tuned independently without any communication. Then, the weights are averaged across nodes at the conclusion of fine-tuning. When fine-tuning DeiT-base and DeiT-large on ImageNet, this procedure matches accuracy in-distribution and improves accuracy under distribution shift compared to the baseline, which observes the same amount of data but communicates gradients at each step. We also observe that lo-fi matches the baseline's performance when fine-tuning OPT language models (up to 1.3B parameters) on Common Crawl. By removing the communication requirement, lo-fi reduces resource barriers for fine-tuning large models and enables fine-tuning in settings with prohibitive communication cost.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 64 of its 80 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

64 of 80 references.

References · 64

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2019Advances and Open Problems in Federated Learningcited by 3,046
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  6. 2019Natural Adversarial Examplescited by 1,408
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  10. 2014How transferable are features in deep neural networks?cited by 1,225
  11. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  12. 2017Mixed Precision Trainingcited by 1,156
  13. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  14. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  15. 2016Deep Networks with Stochastic Depthcited by 979
  16. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  17. 2021Robust fine-tuning of zero-shot modelscited by 797
  18. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  19. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  20. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  21. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  22. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  23. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  24. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  25. 2018Do Better ImageNet Models Transfer Better?cited by 565
  26. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  27. 2022DeiT III: Revenge of the ViTcited by 419
  28. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  29. 2020What is being transferred in transfer learning?cited by 368
  30. 2017Functional Map of the Worldcited by 338
  31. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  32. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  33. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
  34. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  35. 2022Patching open-vocabulary models by interpolating weightscited by 190
  36. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  37. 2018Large scale distributed neural network training through online distillationcited by 146
  38. 2022Federated Learning with Partial Model Personalizationcited by 114

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 13; 2024: 3; 2025: 1.

19 papers.

Sort:

By area:

  1. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  2. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  3. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  4. 2023Deep Model Fusion: A Surveycited by 91
  5. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  6. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  7. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 60
  8. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  9. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  10. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  11. 2023Tangent Model Composition for Ensembling and Continual Fine-tuningcited by 25
  12. 2023Tangent Transformers for Composition, Privacy and Removalcited by 13
7 more
  1. 2023Robot Fleet Learning via Policy Mergingcited by 11
  2. 2023Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classificationcited by 10
  3. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  4. 2023CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Modelcited by 5
  5. 2024PIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory Systemcited by 5
  6. 2025FW-Merging: Scaling Model Merging with Frank-Wolfe Optimizationcited by 5
  7. 2023Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communicationcited by 2