mlsanityData to 7 Oct 2026

PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning

arXiv 2409.16722 ↗First posted 25 Sep 2024cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) and its variants have recently gained much interest due to their ability to avoid excessive inference costs. However, LoRA still encounters the following challenges: (1) Limitation of low-rank assumption; and (2) Its initialization method may be suboptimal. To this end, we propose PMSS(Pre-trained Matrices Skeleton Selection), which enables high-rank updates with low costs while leveraging semantic and linguistic information inherent in pre-trained weight. It achieves this by selecting skeletons from the pre-trained weight matrix and only learning a small matrix instead. Experiments demonstrate that PMSS outperforms LoRA and other fine-tuning methods across tasks with much less trainable parameters. We demonstrate its effectiveness, especially in handling complex tasks such as DROP benchmark(+3.4%/+5.9% on LLaMA2-7B/13B) and math reasoning(+12.89%/+5.61%/+3.11% on LLaMA2-7B, Mistral-7B and Gemma-7B of GSM8K). The code and model will be released soon.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 33 of its 57 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

33 of 57 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
21 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  3. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  4. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  5. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  6. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  7. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  8. 2024LoRA Learns Less and Forgets Lesscited by 313
  9. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  10. 2015Semi-supervised Sequence Learningcited by 198
  11. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  12. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  13. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  14. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  15. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 73
  16. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 57
  17. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  18. 2024QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptationcited by 23
  19. 2020Improving Neural Network Training in Low Dimensional Random Basescited by 19
  20. 2016Greedy Column Subset Selection: New Bounds and Distributed Algorithmscited by 13
  21. 2022Few-Shot Learning by Dimensionality Reduction in Gradient Spacecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025LLM Bias Detection and Mitigation through the Lens of Desired Distributionscited by 4
  2. 2025tCURLoRA: Tensor CUR Decomposition Based Low-Rank Parameter Adaptation and Its Application in Medical Image Segmentationcited by 1
  3. 2025On the Duality between Gradient Transformations and Adapterscited by 1
  4. 2026Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switchingcited by 0
  5. 2026GLT-PEFT: Gated Lie-Tucker Parameter-Efficient Fine-Tuning for Alzheimer's Disease Diagnosis with Hippocampal Segmentation Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
  2. 2026ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
  3. 2025DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
  4. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models
  5. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  6. 2024SBoRA: Low-Rank Adaptation with Regional Weight Updates
  7. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
  8. 2025Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product