# Shears: Unstructured Sparsity with Neural Low-rank Adapter Search

arXiv: https://arxiv.org/abs/2404.10934 · HTML: https://mlsanity.com/paper/2404.10934/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Builds on (most cited first, 19 of 19)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2301.12132/) (2023)
- [MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers](https://mlsanity.com/paper/2109.00799/) (2021)
- [Enabling NAS with Automated Super-Network Generation](https://mlsanity.com/paper/2112.10878/) (2021)

## Built on by (most cited first, 4 of 4)

- [Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models](https://mlsanity.com/paper/2501.17088/) (2025)
- [SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models](https://mlsanity.com/paper/2410.03750/) (2024)
- [Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems](https://mlsanity.com/paper/2502.14305/) (2025)
- [Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets](https://mlsanity.com/paper/2602.14536/) (2026)

## Abstract (arXiv)

Recently, several approaches successfully demonstrated that weight-sharing Neural Architecture Search (NAS) can effectively explore a search space of elastic low-rank adapters (LoRA), allowing the parameter-efficient fine-tuning (PEFT) and compression of large language models. In this paper, we introduce a novel approach called Shears, demonstrating how the integration of cost-effective sparsity and a proposed Neural Low-rank adapter Search (NLS) algorithm can further improve the efficiency of PEFT approaches. Results demonstrate the benefits of Shears compared to other methods, reaching high sparsity levels while improving or with little drop in accuracy, utilizing a single GPU for a pair of hours.