mlsanityData to 14 Oct 2026

ACCEPT: Adaptive Codebook for Composite and Efficient Prompt Tuning

arXiv 2410.12847 ↗First posted 10 Oct 2024cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Prompt Tuning has been a popular Parameter-Efficient Fine-Tuning method attributed to its remarkable performance with few updated parameters on various large-scale pretrained Language Models (PLMs). Traditionally, each prompt has been considered indivisible and updated independently, leading the parameters increase proportionally as prompt length grows. To address this issue, we propose Adaptive Codebook for Composite and Efficient Prompt Tuning (ACCEPT). In our method, we refer to the concept of product quantization (PQ), allowing all soft prompts to share a set of learnable codebook vectors in each subspace, with each prompt differentiated by a set of adaptive weights. We achieve the superior performance on 17 diverse natural language tasks including natural language understanding (NLU) and question answering (QA) tasks by tuning only 0.3% of parameters of the PLMs. Our approach also excels in few-shot and large model settings, highlighting its significant potential.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

39 of 51 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
27 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  5. 2018Neural Network Acceptability Judgmentscited by 958
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  7. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  8. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  9. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  10. 2016NewsQA: A Machine Comprehension Datasetcited by 331
  11. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  12. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  13. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  14. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 225
  15. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  16. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 215
  17. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  18. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 184
  19. 2021Entailment as Few-Shot Learnercited by 123
  20. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  21. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  22. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  23. 2020Extractive Opinion Summarization in Quantized Transformer Spacescited by 76
  24. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
  25. 2018Theory and Experiments on Vector Quantized Autoencoderscited by 55
  26. 2019Unsupervised Paraphrasing without Translationcited by 19
  27. 2021Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoderscited by 19

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Dynamic Task Vector Grouping for Efficient Multi-Task Prompt Tuningcited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2022Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding
  2. 2025A Survey on Prompt Tuning
  3. 2024Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together
  4. 2025PAFT: Prompt-Agnostic Fine-Tuning
  5. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
  6. 2024Efficient Prompting Methods for Large Language Models: A Survey
  7. 2025Fine-tuning for Better Few Shot Prompting: An Empirical Comparison for Short Answer Grading
  8. 2023Few-shot Unified Question Answering: Tuning Models or Prompts?