One-Time Model Adaptation to Heterogeneous Clients: An Intra-Client and Inter-Image Attention Design
Not on the map. Why?
Abstract (arXiv)
The mainstream workflow of image recognition applications is first training one global model on the cloud for a wide range of classes and then serving numerous clients, each with heterogeneous images from a small subset of classes to be recognized. From the cloud-client discrepancies on the range of image classes, the recognition model is desired to have strong adaptiveness, intuitively by concentrating the focus on each individual client's local dynamic class subset, while incurring negligible overhead. In this work, we propose to plug a new intra-client and inter-image attention (ICIIA) module into existing backbone recognition models, requiring only one-time cloud-based training to be client-adaptive. In particular, given a target image from a certain client, ICIIA introduces multi-head self-attention to retrieve relevant images from the client's historical unlabeled images, thereby calibrating the focus and the recognition result. Further considering that ICIIA's overhead is dominated by linear projection, we propose partitioned linear projection with feature shuffling for replacement and allow increasing the number of partitions to dramatically improve efficiency without scarifying too much accuracy. We finally evaluate ICIIA using 3 different recognition tasks with 9 backbone models over 5 representative datasets. Extensive evaluation results demonstrate the effectiveness and efficiency of ICIIA. Specifically, for ImageNet-1K with the backbone models of MobileNetV3-L and Swin-B, ICIIA can improve the testing accuracy to 83.37% (+8.11%) and 88.86% (+5.28%), while adding only 1.62% and 0.02% of FLOPs, respectively.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 56 of its 66 references that are in the map. First posted on arXiv 11 Nov 2022.
Builds on
56 of 66 references.
References · 56
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2016Layer Normalizationcited by 5,579
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2014Deep Learning Face Attributes in the Wildcited by 3,679
- 2022A ConvNet for the 2020scited by 3,545
- 2021Learning to Prompt for Vision-Language Modelscited by 2,269
- 2019Searching for MobileNetV3cited by 1,984
- 2022Visual Prompt Tuningcited by 1,674
- 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
- 2021Video Swin Transformercited by 1,090
- 2018LEAF: A Benchmark for Federated Settingscited by 780
- 2017Federated Multi-Task Learningcited by 669
- 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
- 2020Personalized Federated Learning with Moreau Envelopescited by 612
- 2017Deep Interest Network for Click-Through Rate Predictioncited by 541
- 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
- 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
- 2018Deep Interest Evolution Network for Click-Through Rate Predictioncited by 336
- 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
- 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
- 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
- 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
- 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
- 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
- 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
- 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
- 2022Prompt Distribution Learningcited by 235
- 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
- 2017EMNIST: an extension of MNIST to handwritten letterscited by 212
- 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
- 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
- 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 166
- 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 154
- 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
- 2022Layer-wised Model Aggregation for Personalized Federated Learningcited by 95
50 most cited of 56 All 56 (JSON)
Built on by
Not cited yet.