# Metadata Normalization

arXiv: https://arxiv.org/abs/2104.09052 · HTML: https://mlsanity.com/paper/2104.09052/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 71 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 30 of 30)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [The Variational Fair Autoencoder](https://mlsanity.com/paper/1511.00830/) (2015)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)
- [No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World](https://mlsanity.com/paper/1711.08536/) (2017)
- [Identifying and Correcting Label Bias in Machine Learning](https://mlsanity.com/paper/1901.04966/) (2019)
- [Controllable Invariance through Adversarial Feature Learning](https://mlsanity.com/paper/1705.11122/) (2017)
- [Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition](https://mlsanity.com/paper/1912.05534/) (2019)
- [Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings](https://mlsanity.com/paper/1809.02169/) (2018)
- [Invariant Representations without Adversarial Training](https://mlsanity.com/paper/1805.09458/) (2018)
- [LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning](https://mlsanity.com/paper/1805.09965/) (2018)
- [Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks](https://mlsanity.com/paper/1911.09737/) (2019)
- [Detect and Correct Bias in Multi-Site Neuroimaging Datasets](https://mlsanity.com/paper/2002.05049/) (2020)
- [Adversarial Invariant Feature Learning with Accuracy Constraint for Domain Generalization](https://mlsanity.com/paper/1904.12543/) (2019)
- [Learning Fair Representations for Kernel Models](https://mlsanity.com/paper/1906.11813/) (2019)
- [Confounder-Aware Visualization of ConvNets](https://mlsanity.com/paper/1907.12727/) (2019)
- [AdaBatch: Efficient Gradient Aggregation Rules for Sequential and Parallel Stochastic Gradient Methods](https://mlsanity.com/paper/1711.01761/) (2017)
- [Removing Confounding Factors Associated Weights in Deep Neural Networks Improves the Prediction Accuracy for Healthcare Applications](https://mlsanity.com/paper/1803.07276/) (2018)

## Built on by (most cited first, 14 of 14)

- [Revisiting the "Video" in Video-Language Understanding](https://mlsanity.com/paper/2206.01720/) (2022)
- [Metadata-Conditioned Generative Models to Synthesize Anatomically-Plausible 3D Brain MRIs](https://mlsanity.com/paper/2310.04630/) (2023)
- [Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeat](https://mlsanity.com/paper/2307.05350/) (2023)
- [A Penalty Approach for Normalizing Feature Distributions to Build Confounder-Free Models](https://mlsanity.com/paper/2207.04607/) (2022)
- [DiaMond: Dementia Diagnosis with Multi-Modal Vision Transformers Using MRI and PET](https://mlsanity.com/paper/2410.23219/) (2024)
- [RegBN: Batch Normalization of Multimodal Data with Regularization](https://mlsanity.com/paper/2310.00641/) (2023)
- [Coping with Change: Learning Invariant and Minimum Sufficient Representations for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2306.04893/) (2023)
- [Enforcing Conditional Independence for Fair Representation Learning and Causal Image Generation](https://mlsanity.com/paper/2404.13798/) (2024)
- [Tackling Shortcut Learning in Deep Neural Networks: An Iterative Approach with Interpretable Models](https://mlsanity.com/paper/2302.10289/) (2023)
- [AI-based association analysis for medical imaging using latent-space geometric confounder correction](https://mlsanity.com/paper/2311.12836/) (2023)
- [Generalizing Orthogonalization for Models with Non-Linearities](https://mlsanity.com/paper/2405.02475/) (2024)
- [Confounder-Free Continual Learning via Recursive Feature Normalization](https://mlsanity.com/paper/2507.09031/) (2025)
- [Interpreting Protein Language Model Embeddings via Orthogonal Projection for Protein Fitness Prediction](https://mlsanity.com/paper/2608.25548/) (2026)
- [Controlling for Omitted Variable Bias in Deep Neural Networks](https://mlsanity.com/paper/2608.25930/) (2026)

## Abstract (arXiv)

Batch Normalization (BN) and its variants have delivered tremendous success in combating the covariate shift induced by the training step of deep learning methods. While these techniques normalize feature distributions by standardizing with batch statistics, they do not correct the influence on features from extraneous variables or multiple distributions. Such extra variables, referred to as metadata here, may create bias or confounding effects (e.g., race when classifying gender from face images). We introduce the Metadata Normalization (MDN) layer, a new batch-level operation which can be used end-to-end within the training framework, to correct the influence of metadata on feature distributions. MDN adopts a regression analysis technique traditionally used for preprocessing to remove (regress out) the metadata effects on model features during training. We utilize a metric based on distance correlation to quantify the distribution bias from the metadata and demonstrate that our method successfully removes metadata effects on four diverse settings: one synthetic, one 2D image, one video, and one 3D medical image dataset.