mlsanityData to 7 Oct 2026

Model Balancing Helps Low-data Training and Fine-tuning

arXiv 2410.12178 ↗First posted 16 Oct 2024cs.LG stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent advances in foundation models have emphasized the need to align pre-trained models with specialized domains using small, curated datasets. Studies on these foundation models underscore the importance of low-data training and fine-tuning. This topic, well-known in natural language processing (NLP), has also gained increasing attention in the emerging field of scientific machine learning (SciML). To address the limitations of low-data training and fine-tuning, we draw inspiration from Heavy-Tailed Self-Regularization (HT-SR) theory, analyzing the shape of empirical spectral densities (ESDs) and revealing an imbalance in training quality across different model layers. To mitigate this issue, we adapt a recently proposed layer-wise learning rate scheduler, TempBalance, which effectively balances training quality across layers and enhances low-data training and fine-tuning for both NLP and SciML tasks. Notably, TempBalance demonstrates increasing performance gains as the amount of available tuning data decreases. Comparative analyses further highlight the effectiveness of TempBalance and its adaptability as an "add-on" method for improving model performance.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 49 of its 61 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

49 of 61 references.

References · 49

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
37 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2023LIMA: Less Is More for Alignmentcited by 1,080
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  5. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
  6. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  7. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  8. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  9. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  10. 2023Convolutional Neural Operators for robust and accurate learning of PDEscited by 207
  11. 2022Active Example Selection for In-Context Learningcited by 195
  12. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  13. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  14. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 137
  15. 2019Structural Scaffolds for Citation Intent Classification in Scientific Publicationscited by 129
  16. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 128
  17. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 123
  18. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  19. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  20. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  21. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  22. 2020Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networkscited by 62
  23. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  24. 2019Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networkscited by 43
  25. 2024Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learningcited by 32
  26. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  27. 2024PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Lawscited by 30
  28. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
  29. 2021Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metricscited by 19
  30. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 19
  31. 2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Qualitycited by 18
  32. 2020On 1/n neural representation and robustnesscited by 17
  33. 2020Multiplicative noise and heavy tails in stochastic optimizationcited by 15
  34. 2022On the Overlooked Structure of Stochastic Gradientscited by 13
  35. 2024From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networkscited by 9
  36. 2024Outlier-weighed Layerwise Sampling for LLM Fine-tuningcited by 8
  37. 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  2. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  3. 2025M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Qualitycited by 11
  4. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 11
  5. 2025Eigenspectrum Analysis of Neural Networks without Aspect Ratio Biascited by 9
  6. 2026AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantizationcited by 4
  7. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  8. 2026One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMscited by 2
  9. 2026Layer Collapse in Diffusion Language Modelscited by 1
  10. 2026Unveiling Multi-regime Patterns in SciML: Distinct Failure Modes and Regime-specific Optimizationcited by 1
  11. 2026Spectral Signatures of Large Language Modelscited by 0