Slimming Down LLMs Without Losing Their Minds
Not on the map. Why?
Abstract (arXiv)
This paper investigates and validates the impact of fine-tuning on large language model performance, focusing on parameter-efficient methods (LoRA and QLoRA). We evaluate model capabilities across three key domains: (1) commonsense reasoning (HellaSwag), (2) mathematical reasoning (GSM8K), and (3) multi-domain knowledge (MMLU-CS). Our findings demonstrate that: (1) LoRA-based methods effectively improve task-specific performance while maintaining computational efficiency, and (2) performance strongly depends on alignment between fine-tuning dataset and benchmark tasks. The study provides both theoretical insights into parameter-efficient mechanisms and practical guidance for developers implementing efficient LLM adaptation with limited resources.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 14 references that are in the map. First posted on arXiv 12 Jun 2025.
Builds on
10 of 14 references.
References · 10
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
- 20218-bit Optimizers via Block-wise Quantizationcited by 403
- 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
- 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
- 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
- 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities
- 2025A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios
- 2025CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization
- 2025Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs
- 2025LoRA Is Slower Than You Think
- 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning