# Learning with privileged information via adversarial discriminative modality distillation

arXiv: https://arxiv.org/abs/1810.08437 · HTML: https://mlsanity.com/paper/1810.08437/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2018.

## Built on by (most cited first, 30 of 30)

- [Towards Good Practices for Missing Modality Robust Action Recognition](https://mlsanity.com/paper/2211.13916/) (2022)
- [3D-to-2D Distillation for Indoor Scene Parsing](https://mlsanity.com/paper/2104.02243/) (2021)
- [MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning](https://mlsanity.com/paper/2304.08028/) (2023)
- [Motion-Augmented Self-Training for Video Recognition at Smaller Scale](https://mlsanity.com/paper/2105.01646/) (2021)
- [Robust Multimodal Learning via Representation Decoupling](https://mlsanity.com/paper/2407.04458/) (2024)
- [Toward Understanding Privileged Features Distillation in Learning-to-Rank](https://mlsanity.com/paper/2209.08754/) (2022)
- [AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian Detection](https://mlsanity.com/paper/2405.12944/) (2024)
- [Modality Mixer for Multi-modal Action Recognition](https://mlsanity.com/paper/2208.11314/) (2022)
- [CTDS: Centralized Teacher with Decentralized Student for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2203.08412/) (2022)
- [Exploring Missing Modality in Multimodal Egocentric Datasets](https://mlsanity.com/paper/2401.11470/) (2024)
- [CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems](https://mlsanity.com/paper/2502.17821/) (2025)
- [Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation](https://mlsanity.com/paper/2608.09826/) (2026)
- [One-stage Modality Distillation for Incomplete Multimodal Learning](https://mlsanity.com/paper/2309.08204/) (2023)
- [Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration](https://mlsanity.com/paper/2403.11373/) (2024)
- [Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos](https://mlsanity.com/paper/2404.15161/) (2024)
- [Let it shine: Autofluorescence of Papanicolaou-stain improves AI-based cytological oral cancer detection](https://mlsanity.com/paper/2407.01869/) (2024)
- [Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition](https://mlsanity.com/paper/2408.09035/) (2024)
- [KAN-HyperpointNet for Point Cloud Sequence-Based 3D Human Action Recognition](https://mlsanity.com/paper/2409.09444/) (2024)
- [Multimodal Representation Learning and Fusion](https://mlsanity.com/paper/2506.20494/) (2025)
- [Learning Using Generated Privileged Information by Text-to-Image Diffusion Models](https://mlsanity.com/paper/2309.15238/) (2023)
- [Rethinking Knowledge Transfer in Learning Using Privileged Information](https://mlsanity.com/paper/2408.14319/) (2024)
- [Learning Using Privileged Information for Zero-Shot Action Recognition](https://mlsanity.com/paper/2206.08632/) (2022)
- [Inertial Hallucinations -- When Wearable Inertial Devices Start Seeing Things](https://mlsanity.com/paper/2207.06789/) (2022)
- [Hyperspectral Image Analysis in Single-Modal and Multimodal setting using Deep Learning Techniques](https://mlsanity.com/paper/2403.01546/) (2024)
- [Feature-Supervised Action Modality Transfer](https://mlsanity.com/paper/2108.03329/) (2021)
- [Improving Unimodal Inference with Multimodal Transformers](https://mlsanity.com/paper/2311.10170/) (2023)
- [Modality Mixer Exploiting Complementary Information for Multi-modal Action Recognition](https://mlsanity.com/paper/2311.12344/) (2023)
- [STARS: Shared-specific Translation and Alignment for missing-modality Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2601.17342/) (2026)
- [The Global-Local loop: what is missing in bridging the gap between geospatial data from numerous communities?](https://mlsanity.com/paper/2603.20305/) (2026)
- [Neurosymbolic Imitation Learning with Human Guidance: A Privileged Information Approach](https://mlsanity.com/paper/2605.07166/) (2026)

## Abstract (arXiv)

Heterogeneous data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while training data can be accurately collected to include a variety of sensory modalities, it is often the case that not all of them are available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to extract information from multimodal data in the training stage, in a form that can be exploited at test time, considering limitations such as noisy or missing modalities. This paper presents a new approach in this direction for RGB-D vision tasks, developed within the adversarial learning and privileged information frameworks. We consider the practical case of learning representations from depth and RGB videos, while relying only on RGB data at test time. We propose a new approach to train a hallucination network that learns to distill depth information via adversarial learning, resulting in a clean approach without several losses to balance or hyperparameters. We report state-of-the-art results on object classification on the NYUD dataset and video action recognition on the largest multimodal dataset available for this task, the NTU RGB+D, as well as on the Northwestern-UCLA.