mlsanityData to 7 Oct 2026

Position: Enough of Scaling LLMs! Lets Focus on Downscaling

arXiv 2505.00985 ↗First posted 2 May 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We challenge the dominant focus on neural scaling laws and advocate for a paradigm shift toward downscaling in the development of large language models (LLMs). While scaling laws have provided critical insights into performance improvements through increasing model and dataset size, we emphasize the significant limitations of this approach, particularly in terms of computational inefficiency, environmental impact, and deployment constraints. To address these challenges, we propose a holistic framework for downscaling LLMs that seeks to maintain performance while drastically reducing resource demands. This paper outlines practical strategies for transitioning away from traditional scaling paradigms, advocating for a more sustainable, efficient, and accessible approach to LLM development.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 45 of its 54 references that are in the map. First posted on arXiv 2 May 2025.

Builds on

45 of 54 references.

References · 45

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  6. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  7. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  8. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  11. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  12. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
33 more
  1. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  2. 2023Scaling Data-Constrained Language Modelscited by 391
  3. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  4. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  5. 2021Scaling Laws for Transfercited by 265
  6. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  7. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  8. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  9. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  10. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  11. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  12. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  13. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  14. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  15. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  16. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  17. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  18. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  19. 2022Broken Neural Scaling Lawscited by 102
  20. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  21. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 66
  22. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  23. 2024Bridging the Gap between Different Vocabularies for LLM Ensemblecited by 44
  24. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  25. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  26. 2020On Power Laws in Deep Ensemblescited by 26
  27. 2024Purifying Large Language Models by Ensembling a Small Language Modelcited by 25
  28. 2024Toward Inference-optimal Mixture-of-Expert Large Language Modelscited by 19
  29. 2024ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarioscited by 17
  30. 2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 16
  31. 2023A Hardware Evaluation Framework for Large Language Model Inferencecited by 14
  32. 2023Scaling Laws Do Not Scalecited by 13
  33. 2023The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibrationcited by 4
  2. 2026Small Reward Models via Backward Inferencecited by 2
  3. 2026Selecting Language Models for Social Science: Start Small, Start Open, and Validatecited by 0
  4. 2026Position: Stop Preaching and Start Practising Data Frugality for Responsible Development of AIcited by 0
  5. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0