Deep Companion Learning: Enhancing Generalization Through Historical Consistency
Not on the map. Why?
Abstract (arXiv)
We propose Deep Companion Learning (DCL), a novel training method for Deep Neural Networks (DNNs) that enhances generalization by penalizing inconsistent model predictions compared to its historical performance. To achieve this, we train a deep-companion model (DCM), by using previous versions of the model to provide forecasts on new inputs. This companion model deciphers a meaningful latent semantic structure within the data, thereby providing targeted supervision that encourages the primary model to address the scenarios it finds most challenging. We validate our approach through both theoretical analysis and extensive experimentation, including ablation studies, on a variety of benchmark datasets (CIFAR-100, Tiny-ImageNet, ImageNet-1K) using diverse architectural models (ShuffleNetV2, ResNet, Vision Transformer, etc.), demonstrating state-of-the-art performance.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 26 Jul 2024.
Builds on
36 of 50 references.
References · 36
- 2015Deep Residual Learning for Image Recognitioncited by 51,950
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
- 2015Distilling the Knowledge in a Neural Networkcited by 10,180
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,476
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,991
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,426
- 2016Layer Normalizationcited by 5,624
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,170
24 more
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,730
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,882
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,285
- 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,082
- 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,627
- 2018Group Normalizationcited by 1,518
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,411
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,202
- 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,134
- 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,055
- 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 899
- 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
- 2017Deep Mutual Learningcited by 624
- 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
- 2022Decoupled Knowledge Distillationcited by 382
- 2017Snapshot Ensembles: Train 1, get M for freecited by 350
- 2021Distilling Knowledge via Knowledge Reviewcited by 286
- 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 227
- 2022Knowledge Distillation with the Reused Teacher Classifiercited by 141
- 2020Self-Knowledge Distillation with Progressive Refinement of Targetscited by 106
- 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
- 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
- 2018On the Ineffectiveness of Variance Reduced Optimization for Deep Learningcited by 48
- 2016Recurrent Highway Networkscited by 37
Built on by
Citing papers per year: 2026: 1.
1 paper.