mlsanityData to 7 Oct 2026

Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference

arXiv 2304.04947 ↗First posted 11 Apr 2023cs.CLcited by 69

Not on the map. Why?

Abstract (arXiv)

We propose Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency. CoDA generalizes beyond standard adapter approaches to enable a new way of balancing speed and accuracy using conditional computation. Starting with an existing dense pretrained model, CoDA adds sparse activation together with a small number of new parameters and a light-weight training phase. Our experiments demonstrate that the CoDA approach provides an unexpectedly efficient way to transfer knowledge. Across a variety of language, vision, and speech tasks, CoDA achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approaches with moderate to no accuracy loss and the same parameter efficiency.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 62 of its 81 references that are in the map. First posted on arXiv 11 Apr 2023.

Builds on

62 of 81 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2016Layer Normalizationcited by 5,579
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  13. 2022Token Merging: Your ViT But Fastercited by 995
  14. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  15. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  16. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  17. 2016Sequence-Level Knowledge Distillationcited by 761
  18. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  19. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  20. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  21. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  22. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  23. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  24. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  25. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  26. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  27. 2022Confident Adaptive Language Modelingcited by 297
  28. 2021Hash Layers For Large Sparse Modelscited by 236
  29. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  30. 2015Coordinate Descent Algorithmscited by 218
  31. 2019Structured Pruning of Large Language Modelscited by 201
  32. 2022Transformer Quality in Linear Timecited by 189
  33. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  34. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  35. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  36. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  37. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  38. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 36; 2025: 24; 2026: 4.

69 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  3. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  4. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  5. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  6. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  7. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  8. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  9. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  10. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 29
  11. 2024Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptationcited by 25
  12. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
38 more
  1. 2024Apple Intelligence Foundation Language Modelscited by 22
  2. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
  3. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  4. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  5. 2023Learning to Skip for Language Modelingcited by 19
  6. 2024SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixturecited by 19
  7. 2024A Unified Framework with Multimodal Fine-tuning for Remote Sensing Semantic Segmentationcited by 15
  8. 2025Position: Editing Large Language Models Poses Serious Safety Riskscited by 15
  9. 2025Instruction-Following Pruning for Large Language Modelscited by 14
  10. 2024Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Visioncited by 13
  11. 2024Bayesian-guided Label Mapping for Visual Reprogrammingcited by 12
  12. 2024ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptationcited by 11
  13. 2024Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modulescited by 11
  14. 2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10
  15. 2024RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarizationcited by 10
  16. 2025Re-Imagining Multimodal Instruction Tuning: A Representation Viewcited by 9
  17. 2024BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Modelscited by 7
  18. 2024Skipping Computations in Multimodal LLMscited by 7
  19. 2024Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Modelscited by 7
  20. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  21. 2024Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmonycited by 6
  22. 2024Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injectioncited by 6
  23. 2024LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuningcited by 6
  24. 2025TADFormer : Task-Adaptive Dynamic Transformer for Efficient Multi-Task Learningcited by 5
  25. 2025Revisiting Privacy, Utility, and Efficiency Trade-offs when Fine-Tuning Large Language Modelscited by 5
  26. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  27. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
  28. 2024Introducing Routing Functions to Vision-Language Parameter-Efficient Fine-Tuning with Low-Rank Bottleneckscited by 4
  29. 2024CROME: Cross-Modal Adapters for Efficient Multimodal LLMcited by 4
  30. 2025SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRAcited by 4
  31. 2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
  32. 2023Parameter-Efficient Fine-Tuning with Layer Pruning on Free-Text Sequence-to-Sequence Modelingcited by 3
  33. 2024iConFormer: Dynamic Parameter-Efficient Tuning with Input-Conditioned Adaptationcited by 3
  34. 2024Efficient Fine-Tuning of Single-Cell Foundation Models Enables Zero-Shot Molecular Perturbation Predictioncited by 3
  35. 2025FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Expertscited by 3
  36. 2025Efficient Layer-wise LLM Fine-tuning for Revision Intention Predictioncited by 3
  37. 2025ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Expertscited by 3
  38. 2023Towards Robust and Efficient Continual Language Learningcited by 2

50 most cited of 69 All 69 (JSON)