mlsanityData to 7 Oct 2026

Long-Tailed Visual Recognition via Self-Heterogeneous Integration with Knowledge Excavation

arXiv 2304.01279 ↗First posted 3 Apr 2023cs.CVcited by 22

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Deep neural networks have made huge progress in the last few decades. However, as the real-world data often exhibits a long-tailed distribution, vanilla deep models tend to be heavily biased toward the majority classes. To address this problem, state-of-the-art methods usually adopt a mixture of experts (MoE) to focus on different parts of the long-tailed distribution. Experts in these methods are with the same model depth, which neglects the fact that different classes may have different preferences to be fit by models with different depths. To this end, we propose a novel MoE-based method called Self-Heterogeneous Integration with Knowledge Excavation (SHIKE). We first propose Depth-wise Knowledge Fusion (DKF) to fuse features between different shallow parts and the deep part in one network for each expert, which makes experts more diverse in terms of representation. Based on DKF, we further propose Dynamic Knowledge Transfer (DKT) to reduce the influence of the hardest negative class that has a non-negligible impact on the tail classes in our MoE framework. As a result, the classification accuracy of long-tailed data can be significantly improved, especially for the tail classes. SHIKE achieves the state-of-the-art performance of 56.3%, 60.3%, 75.4%, and 41.9% on CIFAR100-LT (IF100), ImageNet-LT, iNaturalist 2018, and Places-LT, respectively.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 53 of its 71 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

53 of 71 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  8. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  10. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  11. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  12. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
38 more
  1. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  2. 2017Focal Loss for Dense Object Detectioncited by 749
  3. 2019Contrastive Representation Distillationcited by 734
  4. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  5. 2017Deep Mutual Learningcited by 621
  6. 2020Long-tail learning via logit adjustmentcited by 605
  7. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  8. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  9. 2018Born Again Neural Networkscited by 539
  10. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  11. 2019Similarity-Preserving Knowledge Distillationcited by 507
  12. 2019BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognitioncited by 403
  13. 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 397
  14. 2022Decoupled Knowledge Distillationcited by 380
  15. 2019On the Efficacy of Knowledge Distillationcited by 377
  16. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  17. 2019Variational Information Distillation for Knowledge Transfercited by 321
  18. 2020Long-tailed Recognition by Routing Diverse Distribution-Aware Expertscited by 270
  19. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  20. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  21. 2021Parametric Contrastive Learningcited by 217
  22. 2021Distribution Alignment: A Unified Framework for Long-tail Visual Recognitioncited by 198
  23. 2020Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classificationcited by 175
  24. 2020Disentangling Label Distribution for Long-tailed Visual Recognitioncited by 172
  25. 2015Cost Sensitive Learning of Deep Feature Representations from Imbalanced Datacited by 167
  26. 2020Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmaxcited by 152
  27. 2022Balanced Contrastive Learning for Long-Tailed Visual Recognitioncited by 143
  28. 2020M2m: Imbalanced Classification via Major-to-minor Translationcited by 134
  29. 2020The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentationcited by 108
  30. 2022Long-Tailed Recognition via Weight Balancingcited by 107
  31. 2021The Majority Can Help The Minority: Context-rich Minority Oversampling for Long-tailed Classificationcited by 100
  32. 2021MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognitioncited by 99
  33. 2021ResLT: Residual Learning for Long-tailed Recognitioncited by 96
  34. 2021Influence-Balanced Loss for Imbalanced Visual Classificationcited by 91
  35. 2021ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shotcited by 86
  36. 2021FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentationcited by 76
  37. 2021Self Supervision to Distillation for Long-Tailed Visual Recognitioncited by 73
  38. 2022Nested Collaborative Learning for Long-Tailed Visual Recognitioncited by 63

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 5; 2025: 9; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023Fill-Up: Balancing Long-Tailed Data with Generative Modelscited by 34
  2. 2024A Systematic Review on Long-Tailed Learningcited by 27
  3. 2023Feature Fusion from Head to Tail for Long-Tailed Visual Recognitioncited by 20
  4. 2024Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognitioncited by 15
  5. 2025Long-tailed Recognition with Model Rebalancingcited by 10
  6. 2023Adjusting Logit in Gaussian Form for Long-Tailed Visual Recognitioncited by 7
  7. 2023Towards Heterogeneous Long-tailed Learning: Benchmarking, Metrics, and Toolboxcited by 6
  8. 2024Enhanced Long-Tailed Recognition with Contrastive CutMix Augmentationcited by 6
  9. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 6
  10. 2023Long-Tail Learning with Rebalanced Contrastive Losscited by 3
  11. 2024Personalized Federated Learning on Heterogeneous and Long-Tailed Data via Expert Collaborative Learningcited by 1
  12. 2024Text-Guided Mixup Towards Long-Tailed Image Categorizationcited by 1
10 more
  1. 2025Long-tailed Medical Diagnosis with Relation-aware Representation Learning and Iterative Classifier Calibrationcited by 1
  2. 2025Mixture of Balanced Information Bottlenecks for Long-Tailed Visual Recognitioncited by 1
  3. 2025Dual Mixture-of-Experts Framework for Discrete-Time Survival Analysiscited by 1
  4. 2023Long-Tailed Classification Based on Coarse-Grained Leading Forest and Multi-Center Losscited by 0
  5. 2025PI-H2T: Enhancing Long-Tailed Visual Recognition with Permutation-Invariant and Head-to-Tail Feature Fusioncited by 0
  6. 2025Long-Tailed Learning for Generalized Category Discoverycited by 0
  7. 2025Unsupervised Contrastive Learning Using Out-Of-Distribution Data for Long-Tailed Datasetcited by 0
  8. 2025Exploring "Many in Few" and "Few in Many" Properties in Long-Tailed, Highly-Imbalanced IC Defect Classificationcited by 0
  9. 2026Balancing Knowledge Distillation for Imbalance Learning with Bilevel Optimizationcited by 0
  10. 2026VICAL: Vicinal Consistency Alignment for Long-Tailed Visual Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Divide, Weight, and Route: Difficulty-Aware Optimization with Dynamic Expert Fusion for Long-tailed Recognition
  2. 2023From Sparse to Soft Mixtures of Experts
  3. 2024Mixture of Experts in Image Classification: What's the Sweet Spot?
  4. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  5. 2026Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement
  6. 2021Scaling Vision with Sparse Mixture of Experts
  7. 2025Enhancing Features in Long-tailed Data Using Large Vision Model
  8. 2025Efficient Long-Tail Learning in Latent Space by sampling Synthetic Data