# Generalizing Orthogonalization for Models with Non-Linearities

arXiv: https://arxiv.org/abs/2405.02475 · HTML: https://mlsanity.com/paper/2405.02475/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 3 May 2024.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Generalizing Across Domains via Cross-Gradient Training](https://mlsanity.com/paper/1804.10745/) (2018)
- [A Convex Framework for Fair Regression](https://mlsanity.com/paper/1706.02409/) (2017)
- [Fair Kernel Learning](https://mlsanity.com/paper/1710.05578/) (2017)
- [Controllable Orthogonalization in Training DNNs](https://mlsanity.com/paper/2004.00917/) (2020)
- [Risk of Bias in Chest Radiography Deep Learning Foundation Models](https://mlsanity.com/paper/2209.02965/) (2022)
- [Fair Infinitesimal Jackknife: Mitigating the Influence of Biased Training Data Points Without Refitting](https://mlsanity.com/paper/2212.06803/) (2022)
- [Semi-Structured Distributional Regression -- Extending Structured Additive Models by Arbitrary Deep Neural Networks and Data Modalities](https://mlsanity.com/paper/2002.05777/) (2020)
- [Understanding Fairness of Gender Classification Algorithms Across Gender-Race Groups](https://mlsanity.com/paper/2009.11491/) (2020)
- [Deep Conditional Transformation Models](https://mlsanity.com/paper/2010.07860/) (2020)
- [Metadata Normalization](https://mlsanity.com/paper/2104.09052/) (2021)
- [Semi-Structured Deep Piecewise Exponential Models](https://mlsanity.com/paper/2011.05824/) (2020)
- [Achieving Fairness with a Simple Ridge Penalty](https://mlsanity.com/paper/2105.13817/) (2021)
- [Controlling Directions Orthogonal to a Classifier](https://mlsanity.com/paper/2201.11259/) (2022)
- [Fair Generalized Linear Models with a Convex Penalty](https://mlsanity.com/paper/2206.09076/) (2022)
- [A New PHO-rmula for Improved Performance of Semi-Structured Networks](https://mlsanity.com/paper/2306.00522/) (2023)
- [Project and Probe: Sample-Efficient Domain Adaptation by Interpolating Orthogonal Features](https://mlsanity.com/paper/2302.05441/) (2023)
- [A Penalty Approach for Normalizing Feature Distributions to Build Confounder-Free Models](https://mlsanity.com/paper/2207.04607/) (2022)
- [Scalable Infomin Learning](https://mlsanity.com/paper/2302.10701/) (2023)
- [Uncertainty-aware predictive modeling for fair data-driven decisions](https://mlsanity.com/paper/2211.02730/) (2022)
- [fairml: A Statistician's Take on Fair Machine Learning Modelling](https://mlsanity.com/paper/2305.02009/) (2023)
- [Learning Fair and Interpretable Representations via Linear Orthogonalization](https://mlsanity.com/paper/1910.12854/) (2019)

## Abstract (arXiv)

The complexity of black-box algorithms can lead to various challenges, including the introduction of biases. These biases present immediate risks in the algorithms' application. It was, for instance, shown that neural networks can deduce racial information solely from a patient's X-ray scan, a task beyond the capability of medical experts. If this fact is not known to the medical expert, automatic decision-making based on this algorithm could lead to prescribing a treatment (purely) based on racial information. While current methodologies allow for the "orthogonalization" or "normalization" of neural networks with respect to such information, existing approaches are grounded in linear models. Our paper advances the discourse by introducing corrections for non-linearities such as ReLU activations. Our approach also encompasses scalar and tensor-valued predictions, facilitating its integration into neural network architectures. Through extensive experiments, we validate our method's effectiveness in safeguarding sensitive data in generalized linear models, normalizing convolutional neural networks for metadata, and rectifying pre-existing embeddings for undesired attributes.