# Deep Supervision with Intermediate Concepts

arXiv: https://arxiv.org/abs/1801.03399 · HTML: https://mlsanity.com/paper/1801.03399/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2018.

## Built on by (most cited first, 19 of 19)

- [Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off](https://mlsanity.com/paper/2209.09056/) (2022)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [Towards Learning Food Portion From Monocular Images With Cross-Domain Feature Adaptation](https://mlsanity.com/paper/2103.07562/) (2021)
- [Timestamp-Supervised Action Segmentation with Graph Convolutional Networks](https://mlsanity.com/paper/2206.15031/) (2022)
- [Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment](https://mlsanity.com/paper/2305.19478/) (2023)
- [HS3: Learning with Proper Task Complexity in Hierarchically Supervised Semantic Segmentation](https://mlsanity.com/paper/2111.02333/) (2021)
- [A Comprehensive Review on Deep Supervision: Theories and Applications](https://mlsanity.com/paper/2207.02376/) (2022)
- [A Survey of Neural Trees](https://mlsanity.com/paper/2209.03415/) (2022)
- [Action Segmentation Using 2D Skeleton Heatmaps and Multi-Modality Fusion](https://mlsanity.com/paper/2309.06462/) (2023)
- [Interpretable Prognostics with Concept Bottleneck Models](https://mlsanity.com/paper/2405.17575/) (2024)
- [Blocks World Revisited: The Effect of Self-Occlusion on Classification by Convolutional Neural Networks](https://mlsanity.com/paper/2102.12911/) (2021)
- [Parallel Multi-Scale Networks with Deep Supervision for Hand Keypoint Detection](https://mlsanity.com/paper/2112.10275/) (2021)
- [Progressive DeepSSM: Training Methodology for Image-To-Shape Deep Models](https://mlsanity.com/paper/2310.01529/) (2023)
- [Rethinking Boundary Detection in Deep Learning-Based Medical Image Segmentation](https://mlsanity.com/paper/2505.04652/) (2025)
- [Multi-channel Deep Supervision for Crowd Counting](https://mlsanity.com/paper/2103.09553/) (2021)
- [MultiScale Probability Map guided Index Pooling with Attention-based learning for Road and Building Segmentation](https://mlsanity.com/paper/2302.09411/) (2023)
- [WTTFNet: A Weather-Time-Trajectory Fusion Network for Pedestrian Trajectory Prediction in Urban Complex](https://mlsanity.com/paper/2405.18945/) (2024)
- [3DPX: Single Panoramic X-ray Analysis Guided by 3D Oral Structure Reconstruction](https://mlsanity.com/paper/2409.18701/) (2024)
- [MDSkin-Net: Multi-Task Skin Lesion Analysis Driven by Pattern Analysis Priors and Spatial Alignment Regularization](https://mlsanity.com/paper/2609.30855/) (2026)

## Abstract (arXiv)

Recent data-driven approaches to scene interpretation predominantly pose inference as an end-to-end black-box mapping, commonly performed by a Convolutional Neural Network (CNN). However, decades of work on perceptual organization in both human and machine vision suggests that there are often intermediate representations that are intrinsic to an inference task, and which provide essential structure to improve generalization. In this work, we explore an approach for injecting prior domain structure into neural network training by supervising hidden layers of a CNN with intermediate concepts that normally are not observed in practice. We formulate a probabilistic framework which formalizes these notions and predicts improved generalization via this deep supervision method. One advantage of this approach is that we are able to train only from synthetic CAD renderings of cluttered scenes, where concept values can be extracted, but apply the results to real images. Our implementation achieves the state-of-the-art performance of 2D/3D keypoint localization and image classification on real image benchmarks, including KITTI, PASCAL VOC, PASCAL3D+, IKEA, and CIFAR100. We provide additional evidence that our approach outperforms alternative forms of supervision, such as multi-task networks.