# Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing

arXiv: https://arxiv.org/abs/2210.04153 · HTML: https://mlsanity.com/paper/2210.04153/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 9 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [DropBlock: A regularization method for convolutional networks](https://mlsanity.com/paper/1810.12890/) (2018)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018)
- [PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://mlsanity.com/paper/1805.04409/) (2018)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [The Shattered Gradients Problem: If resnets are the answer, then what is the question?](https://mlsanity.com/paper/1702.08591/) (2017)
- [β-DARTS: Beta-Decay Regularization for Differentiable Architecture Search](https://mlsanity.com/paper/2203.01665/) (2022)
- [Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation](https://mlsanity.com/paper/2206.06067/) (2022)
- [Improving One-shot NAS by Suppressing the Posterior Fading](https://mlsanity.com/paper/1910.02543/) (2019)
- [CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deployment](https://mlsanity.com/paper/2104.12642/) (2021)
- [BN-NAS: Neural Architecture Search with Batch Normalization](https://mlsanity.com/paper/2108.07375/) (2021)
- [Polynomial Regression As an Alternative to Neural Nets](https://mlsanity.com/paper/1806.06850/) (2018)
- [Efficient Joint-Dimensional Search with Solution Space Regularization for Real-Time Semantic Segmentation](https://mlsanity.com/paper/2208.05271/) (2022)

## Built on by (most cited first, 10 of 10)

- [EMR-Merging: Tuning-Free High-Performance Model Merging](https://mlsanity.com/paper/2405.17461/) (2024)
- [ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentation](https://mlsanity.com/paper/2401.12665/) (2024)
- [β-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search](https://mlsanity.com/paper/2301.06393/) (2023)
- [Partial Fine-Tuning: A Successor to Full Fine-Tuning for Vision Transformers](https://mlsanity.com/paper/2312.15681/) (2023)
- [Stimulative Training++: Go Beyond The Performance Limits of Residual Networks](https://mlsanity.com/paper/2305.02507/) (2023)
- [Boosting Residual Networks with Group Knowledge](https://mlsanity.com/paper/2308.13772/) (2023)
- [Enhanced Sparsification via Stimulative Training](https://mlsanity.com/paper/2403.06417/) (2024)
- [Multi-view Vision-Prompt Fusion Network: Can 2D Pre-trained Model Boost 3D Point Cloud Data-scarce Learning?](https://mlsanity.com/paper/2304.10224/) (2023)
- [S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning](https://mlsanity.com/paper/2410.07046/) (2024)
- [PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding](https://mlsanity.com/paper/2506.17310/) (2025)

## Abstract (arXiv)

Residual networks have shown great success and become indispensable in today's deep models. In this work, we aim to re-investigate the training process of residual networks from a novel social psychology perspective of loafing, and further propose a new training strategy to strengthen the performance of residual networks. As residual networks can be viewed as ensembles of relatively shallow networks (i.e., textitunraveled view) in prior works, we also start from such view and consider that the final performance of a residual network is co-determined by a group of sub-networks. Inspired by the social loafing problem of social psychology, we find that residual networks invariably suffer from similar problem, where sub-networks in a residual network are prone to exert less effort when working as part of the group compared to working alone. We define this previously overlooked problem as textitnetwork loafing. As social loafing will ultimately cause the low individual productivity and the reduced overall performance, network loafing will also hinder the performance of a given residual network and its sub-networks. Referring to the solutions of social psychology, we propose textitstimulative training, which randomly samples a residual sub-network and calculates the KL-divergence loss between the sampled sub-network and the given residual network, to act as extra supervision for sub-networks and make the overall goal consistent. Comprehensive empirical results and theoretical analyses verify that stimulative training can well handle the loafing problem, and improve the performance of a residual network by improving the performance of its sub-networks. The code is available at https://github.com/Sunshine-Ye/NIPS22-ST .