# Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives

arXiv: https://arxiv.org/abs/2003.10739 · HTML: https://mlsanity.com/paper/2003.10739/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2020.

## Built on by (most cited first, 4 of 4)

- [Contrastive Deep Supervision](https://mlsanity.com/paper/2207.05306/) (2022)
- [TiCoSS: Tightening the Coupling between Semantic Segmentation and Stereo Matching within A Joint Learning Framework](https://mlsanity.com/paper/2407.18038/) (2024)
- [Contextual Classification Using Self-Supervised Auxiliary Models for Deep Neural Networks](https://mlsanity.com/paper/2101.03057/) (2021)
- [Deeply Supervised Layer Selective Attention Network: Towards Label-Efficient Learning for Medical Image Classification](https://mlsanity.com/paper/2209.13844/) (2022)

## Abstract (arXiv)

While the depth of modern Convolutional Neural Networks (CNNs) surpasses that of the pioneering networks with a significant margin, the traditional way of appending supervision only over the final classifier and progressively propagating gradient flow upstream remains the training mainstay. Seminal Deeply-Supervised Networks (DSN) were proposed to alleviate the difficulty of optimization arising from gradient flow through a long chain. However, it is still vulnerable to issues including interference to the hierarchical representation generation process and inconsistent optimization objectives, as illustrated theoretically and empirically in this paper. Complementary to previous training strategies, we propose Dynamic Hierarchical Mimicking, a generic feature learning mechanism, to advance CNN training with enhanced generalization ability. Partially inspired by DSN, we fork delicately designed side branches from the intermediate layers of a given neural network. Each branch can emerge from certain locations of the main branch dynamically, which not only retains representation rooted in the backbone network but also generates more diverse representations along its own pathway. We go one step further to promote multi-level interactions among different branches through an optimization formula with probabilistic prediction matching losses, thus guaranteeing a more robust optimization process and better representation ability. Experiments on both category and instance recognition tasks demonstrate the substantial improvements of our proposed method over its corresponding counterparts using diverse state-of-the-art CNN architectures. Code and models are publicly available at https://github.com/d-li14/DHM