mlsanityData to 14 Oct 2026

Shears: Unstructured Sparsity with Neural Low-rank Adapter Search

arXiv 2404.10934 ↗First posted 16 Apr 2024cs.LG cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Recently, several approaches successfully demonstrated that weight-sharing Neural Architecture Search (NAS) can effectively explore a search space of elastic low-rank adapters (LoRA), allowing the parameter-efficient fine-tuning (PEFT) and compression of large language models. In this paper, we introduce a novel approach called Shears, demonstrating how the integration of cost-effective sparsity and a proposed Neural Low-rank adapter Search (NLS) algorithm can further improve the efficiency of PEFT approaches. Results demonstrate the benefits of Shears compared to other methods, reaching high sparsity levels while improving or with little drop in accuracy, utilizing a single GPU for a pair of hours.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

19 of 30 references.

References · 19

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  10. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  11. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  12. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
7 more
  1. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  2. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  3. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 188
  4. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  5. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  6. 2021MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solverscited by 39
  7. 2021Enabling NAS with Automated Super-Network Generationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Modelscited by 7
  2. 2024SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Modelscited by 5
  3. 2025Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systemscited by 2
  4. 2026Explainable Token-level Noise Filtering for LLM Fine-tuning Datasetscited by 1