# NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuning

arXiv: https://arxiv.org/abs/2510.18940 · HTML: https://mlsanity.com/paper/2510.18940/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Builds on (most cited first, 30 of 34)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks](https://mlsanity.com/paper/2112.06825/) (2021)
- [VeRA: Vector-based Random Matrix Adaptation](https://mlsanity.com/paper/2310.11454/) (2023)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [ReFT: Representation Finetuning for Language Models](https://mlsanity.com/paper/2404.03592/) (2024)
- [Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2004.03829/) (2020)
- [CoIR: A Comprehensive Benchmark for Code Information Retrieval Models](https://mlsanity.com/paper/2407.02883/) (2024)
- [Parameter-Efficient Fine-Tuning without Introducing New Latency](https://mlsanity.com/paper/2305.16742/) (2023)
- [MMDocIR: Benchmarking Multimodal Retrieval for Long Documents](https://mlsanity.com/paper/2501.08828/) (2025)
- [Advancing Parameter Efficiency in Fine-tuning via Representation Editing](https://mlsanity.com/paper/2402.15179/) (2024)
- [Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering](https://mlsanity.com/paper/2505.16470/) (2025)

## Abstract (arXiv)

Existing parameter-efficient fine-tuning (PEFT) methods primarily fall into two categories: addition-based and selective in-situ adaptation. The former, such as LoRA, introduce additional modules to adapt the model to downstream tasks, offering strong memory efficiency. However, their representational capacity is often limited, making them less suitable for fine-grained adaptation. In contrast, the latter directly fine-tunes a carefully chosen subset of the original model parameters, allowing for more precise and effective adaptation, but at the cost of significantly increased memory consumption. To reconcile this trade-off, we propose NeuroAda, a novel PEFT method that enables fine-grained model finetuning while maintaining high memory efficiency. Our approach first identifies important parameters (i.e., connections within the network) as in selective adaptation, and then introduces bypass connections for these selected parameters. During finetuning, only the bypass connections are updated, leaving the original model parameters frozen. Empirical results on 23+ tasks spanning both natural language generation and understanding demonstrate that NeuroAda achieves state-of-the-art performance with as little as ≤ 0.02% trainable parameters, while reducing CUDA memory usage by up to 60%. We release our code here: https://github.com/FightingFighting/NeuroAda.git.