mlsanityData to 7 Oct 2026

Optimize Incompatible Parameters through Compatibility-aware Knowledge Integration

arXiv 2501.07596 ↗First posted 10 Jan 2025cs.LG cs.CL cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep neural networks have become foundational to advancements in multiple domains, including recommendation systems, natural language processing, and so on. Despite their successes, these models often contain incompatible parameters that can be underutilized or detrimental to model performance, particularly when faced with specific, varying data distributions. Existing research excels in removing such parameters or merging the outputs of multiple different pretrained models. However, the former focuses on efficiency rather than performance, while the latter requires several times more computing and storage resources to support inference. In this paper, we set the goal to explicitly improve these incompatible parameters by leveraging the complementary strengths of different models, thereby directly enhancing the models without any additional parameters. Specifically, we propose Compatibility-aware Knowledge Integration (CKI), which consists of Parameter Compatibility Assessment and Parameter Splicing, which are used to evaluate the knowledge content of multiple models and integrate the knowledge into one model, respectively. The integrated model can be used directly for inference or for further fine-tuning. We conduct extensive experiments on various datasets for recommendation and language tasks, and the results show that Compatibility-aware Knowledge Integration can effectively optimize incompatible parameters under multiple tasks and settings to break through the training limit of the original model without increasing the inference cost.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

36 of 50 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2019Searching for MobileNetV3cited by 1,984
  10. 2018Self-Attentive Sequential Recommendationcited by 1,169
  11. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,134
  12. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
24 more
  1. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 952
  2. 2015Session-based Recommendations with Recurrent Neural Networkscited by 685
  3. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 673
  4. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  6. 2017Deep Interest Network for Click-Through Rate Predictioncited by 541
  7. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  8. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  9. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 189
  10. 2020Domain Adaptive Ensemble Learningcited by 156
  11. 2021FairFed: Enabling Group Fairness in Federated Learningcited by 114
  12. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  13. 2022Intent Contrastive Learning for Sequential Recommendationcited by 90
  14. 2019Pruning from Scratchcited by 72
  15. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 66
  16. 2024Efficient Tuning and Inference for Large Language Models on Textual Graphscited by 54
  17. 2020Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computingcited by 52
  18. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  19. 2023Personalized Federated Learning under Mixture of Distributionscited by 37
  20. 2022DUET: A Tuning-Free Device-Cloud Collaborative Parameters Generation Framework for Efficient Device Model Generalizationcited by 35
  21. 2023Intelligent Model Update Strategy for Sequential Recommendationcited by 25
  22. 2023A Three-regime Model of Network Pruningcited by 15
  23. 2024Bridging Local Details and Global Context in Text-Attributed Graphscited by 15
  24. 2024Semantic Codebook Learning for Dynamic Recommendation Modelscited by 6

Built on by

Not cited yet.