mlsanityData to 14 Oct 2026

PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers

arXiv 2410.01870 ↗First posted 2 Oct 2024cs.LG cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large pre-trained foundation models often yields excellent downstream performance but is prohibitively expensive when updating all parameters. Parameter-efficient fine-tuning (PEFT) methods such as LoRA alleviate this by introducing lightweight update modules, yet they commonly rely on weight-agnostic linear approximations, limiting their expressiveness. In this work, we propose PEANuT, a novel PEFT framework that introduces weight-aware neural tweakers, compact neural modules that generate task-adaptive updates conditioned on frozen pre-trained weights. PEANuT provides a flexible yet efficient way to capture complex update patterns without full model tuning. We theoretically show that PEANuT achieves equivalent or greater expressivity than existing linear PEFT methods with comparable or fewer parameters. Extensive experiments across four benchmarks with over twenty datasets demonstrate that PEANuT consistently outperforms strong baselines in both NLP and vision tasks, while maintaining low computational overhead.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 67 of its 89 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

67 of 89 references.

References · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  13. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  14. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  15. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  16. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  17. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  18. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  19. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  20. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  21. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  22. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  23. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  24. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  25. 2016On the Expressive Power of Deep Neural Networkscited by 306
  26. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  27. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  28. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  29. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  30. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  31. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  32. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  33. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  34. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  35. 2022Neural Prompt Searchcited by 135
  36. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  37. 2023Text Classification via Large Language Modelscited by 106
  38. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 3.

8 papers.

  1. 2024Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimizationcited by 26
  2. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  3. 2026HealthMamba: An Uncertainty-aware Spatiotemporal Graph State Space Model for Effective and Reliable Healthcare Facility Visit Predictioncited by 8
  4. 2025DiaBlo: Diagonal Blocks Are Sufficient For Finetuningcited by 2
  5. 2025Enhancing Low-Rank Adaptation with Structured Nonlinear Transformationscited by 2
  6. 2024Selective State Space Memory for Large Vision-Language Modelscited by 0
  7. 2025Optimizing Retrieval-Augmented Generation (RAG) for Colloquial Cantonese: A LoRA-Based Systematic Reviewcited by 0
  8. 2026CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation with Non-linearity Retained at Inferencecited by 0