Data to 7 Oct 2026

Efficient parametrization of multi-domain deep neural networks

arXiv 1803.10082 ↗First posted 27 Mar 2018cs.CV stat.MLcited by 168

Not on the map. Why?

Abstract (arXiv)

A practical limitation of deep neural networks is their high degree of specialization to a single task and visual domain. Recently, inspired by the successes of transfer learning, several authors have proposed to learn instead universal, fixed feature extractors that, used as the first stage of any deep network, work well for several tasks and domains simultaneously. Nevertheless, such universal features are still somewhat inferior to specialized networks. To overcome this limitation, in this paper we propose to consider instead universal parametric families of neural networks, which still contain specialized problem-specific models, but differing only by a small number of parameters. We study different designs for such parametrizations, including series and parallel residual adapters, joint adapter compression, and parameter allocations, and empirically identify the ones that yield the highest compression. We show that, in order to maximize performance, it is necessary to adapt both shallow and deep layers of a deep network, but the required changes are very small. We also show that these universal parametrization are very effective for transfer learning, where they outperform traditional fine-tuning techniques.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 44; 2023: 47; 2024: 28; 2025: 15; 2026: 3.

168 papers.

Sort:

By area:

2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
2022Visual Prompt Tuningcited by 1,674
2022Vision Transformer Adapter for Dense Predictionscited by 564
2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
2021Sustainable Modular Debiasing of Language Modelscited by 98

38 more

2022Three things everyone should know about Vision Transformerscited by 90
2023Modular Deep Learningcited by 82
2021Cross-domain Few-shot Learning with Task-specific Adapterscited by 79
2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
2022Real-Time Style Modelling of Human Locomotion via Feature-Wise Transformations and Local Motion Phasescited by 71
2022Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNetscited by 62
2023ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptationcited by 58
2022Contextual Adapters for Personalized Speech Recognition in Neural Transducerscited by 57
2021Universal Representation Learning from Multiple Domains for Few-shot Classificationcited by 56
2021Self-Supervised Training Enhances Online Continual Learningcited by 53
2022Pro-tuning: Unified Prompt Tuning for Vision Taskscited by 53
2022Generalizing to New Physical Systems via Context-Informed Dynamics Modelcited by 49
2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
2023Annotation-free Audio-Visual Segmentationcited by 44
2021Cluster-driven Graph Federated Learning over Multiple Domainscited by 43
2022Task Adaptive Parameter Sharing for Multi-Task Learningcited by 42
2021A linearized framework and a new benchmark for model selection for fine-tuningcited by 41
2022KG-SP: Knowledge Guided Simple Primitives for Open World Compositional Zero-Shot Learningcited by 41
2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40
2022Multi-domain Learning for Updating Face Anti-spoofing Modelscited by 39
2021Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safetycited by 35
2023Visual Tuningcited by 34
2021Learning Multiple Stock Trading Patterns with Temporal Routing Adaptor and Optimal Transportcited by 32
2024How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Modelcited by 32
2023Towards Flexible Multi-modal Document Modelscited by 30
2023ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systemscited by 28
2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
2024CAT-SAM: Conditional Tuning for Few-Shot Adaptation of Segment Anything Modelcited by 27
2023CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generationcited by 26
2021Multi-Domain Incremental Learning for Semantic Segmentationcited by 25
2023mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinationscited by 24
2022Universal Representations: A Unified Look at Multiple Task and Domain Learningcited by 23
2022Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfercited by 22
2021Unsupervised Domain Adaptation in LiDAR Semantic Segmentation with Self-Supervision and Gated Adapterscited by 21
2023Learning Expressive Prompting With Residuals for Vision Transformerscited by 21
2022Proper Reuse of Image Classification Features Improves Object Detectioncited by 20
2024Context-based and Diversity-driven Specificity in Compositional Zero-Shot Learningcited by 20

50 most cited of 168 All 168 (JSON)