# Encoding Hierarchical Information in Neural Networks helps in Subpopulation Shift

arXiv: https://arxiv.org/abs/2112.10844 · HTML: https://mlsanity.com/paper/2112.10844/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 73 references that are in the map.
- First posted on arXiv 20 Dec 2021.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Optimal Transport for Domain Adaptation](https://mlsanity.com/paper/1507.00504/) (2015)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Domain Generalization for Object Recognition with Multi-task Autoencoders](https://mlsanity.com/paper/1508.07680/) (2015)
- [BREEDS: Benchmarks for Subpopulation Shift](https://mlsanity.com/paper/2008.04859/) (2020)
- [Understanding Self-Training for Gradual Domain Adaptation](https://mlsanity.com/paper/2002.11361/) (2020)
- [Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift](https://mlsanity.com/paper/2003.04475/) (2020)
- [Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks](https://mlsanity.com/paper/1912.09393/) (2019)
- [Towards a Theoretical Framework of Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.04496/) (2021)
- [Interpreting CNNs via Decision Trees](https://mlsanity.com/paper/1802.00121/) (2018)
- [Domain-Adversarial Neural Networks](https://mlsanity.com/paper/1412.4446/) (2014)
- [Collaborative Learning for Deep Neural Networks](https://mlsanity.com/paper/1805.11761/) (2018)
- [Self-training Avoids Using Spurious Features Under Domain Shift](https://mlsanity.com/paper/2006.10032/) (2020)
- [Interpretable Image Recognition with Hierarchical Prototypes](https://mlsanity.com/paper/1906.10651/) (2019)
- [Do Convolutional Neural Networks Learn Class Hierarchy?](https://mlsanity.com/paper/1710.06501/) (2017)
- [Interpreting Chest X-rays via CNNs that Exploit Hierarchical Disease Dependencies and Uncertainty Labels](https://mlsanity.com/paper/2005.12734/) (2020)
- [A Theory of Label Propagation for Subpopulation Shift](https://mlsanity.com/paper/2102.11203/) (2021)
- [Hierarchical Image Classification using Entailment Cone Embeddings](https://mlsanity.com/paper/2004.03459/) (2020)
- [Hierarchy-based Image Embeddings for Semantic Image Retrieval](https://mlsanity.com/paper/1809.09924/) (2018)

## Built on by (most cited first, 3 of 3)

- [Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations](https://mlsanity.com/paper/2404.12698/) (2024)
- [Diverse Prototypical Ensembles Improve Robustness to Subpopulation Shift](https://mlsanity.com/paper/2505.23027/) (2025)
- [Adaptive Multi-Agent Feature Selection for Personalized Fall Risk Prevention](https://mlsanity.com/paper/2608.18450/) (2026)

## Abstract (arXiv)

Over the past decade, deep neural networks have proven to be adept in image classification tasks, often surpassing humans in terms of accuracy. However, standard neural networks often fail to understand the concept of hierarchical structures and dependencies among different classes for vision related tasks. Humans on the other hand, seem to intuitively learn categories conceptually, progressively growing from understanding high-level concepts down to granular levels of categories. One of the issues arising from the inability of neural networks to encode such dependencies within its learned structure is that of subpopulation shift -- where models are queried with novel unseen classes taken from a shifted population of the training set categories. Since the neural network treats each class as independent from all others, it struggles to categorize shifting populations that are dependent at higher levels of the hierarchy. In this work, we study the aforementioned problems through the lens of a novel conditional supervised training framework. We tackle subpopulation shift by a structured learning procedure that incorporates hierarchical information conditionally through labels. Furthermore, we introduce a notion of graphical distance to model the catastrophic effect of mispredictions. We show that learning in this structured hierarchical manner results in networks that are more robust against subpopulation shifts, with an improvement up to 3% in terms of accuracy and up to 11% in terms of graphical distance over standard models on subpopulation shift benchmarks.