# A New PHO-rmula for Improved Performance of Semi-Structured Networks

arXiv: https://arxiv.org/abs/2306.00522 · HTML: https://mlsanity.com/paper/2306.00522/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Purifying Interaction Effects with the Functional ANOVA: An Efficient Algorithm for Recovering Identifiable Additive Models](https://mlsanity.com/paper/1911.04974/) (2019)
- [Semi-Structured Distributional Regression -- Extending Structured Additive Models by Arbitrary Deep Neural Networks and Data Modalities](https://mlsanity.com/paper/2002.05777/) (2020)
- [Deep Conditional Transformation Models](https://mlsanity.com/paper/2010.07860/) (2020)
- [Deep transformation models: Tackling complex regression problems with neural network based transformation models](https://mlsanity.com/paper/2004.00464/) (2020)
- [Semi-Structured Deep Piecewise Exponential Models](https://mlsanity.com/paper/2011.05824/) (2020)
- [deepregression: a Flexible Neural Network Framework for Semi-Structured Deep Distributional Regression](https://mlsanity.com/paper/2104.02705/) (2021)
- [A Wide and Deep Neural Network for Survival Analysis from Anatomical Shape and Tabular Clinical Data](https://mlsanity.com/paper/1909.03890/) (2019)
- [DeepPAMM: Deep Piecewise Exponential Additive Mixed Models for Complex Hazard Structures in Survival Analysis](https://mlsanity.com/paper/2202.07423/) (2022)
- [Combining Graph Neural Networks and Spatio-temporal Disease Models to Predict COVID-19 Cases in Germany](https://mlsanity.com/paper/2101.00661/) (2021)
- [Deep and interpretable regression models for ordinal outcomes](https://mlsanity.com/paper/2010.08376/) (2020)
- [Factorized Structured Regression for Large-Scale Varying Coefficient Models](https://mlsanity.com/paper/2205.13080/) (2022)
- [Deep Bayesian regression models](https://mlsanity.com/paper/1806.02160/) (2018)

## Built on by (most cited first, 11 of 11)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Scalable Higher-Order Tensor Product Spline Models](https://mlsanity.com/paper/2402.01090/) (2024)
- [Bayesian Semi-structured Subspace Inference](https://mlsanity.com/paper/2401.12950/) (2024)
- [How Inverse Conditional Flows Can Serve as a Substitute for Distributional Regression](https://mlsanity.com/paper/2405.05429/) (2024)
- [Post-hoc Orthogonalization for Mitigation of Protected Feature Bias in CXR Embeddings](https://mlsanity.com/paper/2311.01349/) (2023)
- [Generalizing Orthogonalization for Models with Non-Linearities](https://mlsanity.com/paper/2405.02475/) (2024)
- [A Functional Extension of Semi-Structured Networks](https://mlsanity.com/paper/2410.05430/) (2024)
- [LiD-GLM: Lipschitz-constrained Deep Generalized Linear Models](https://mlsanity.com/paper/2608.16340/) (2026)
- [Interpreting Protein Language Model Embeddings via Orthogonal Projection for Protein Fitness Prediction](https://mlsanity.com/paper/2608.25548/) (2026)
- [Controlling for Omitted Variable Bias in Deep Neural Networks](https://mlsanity.com/paper/2608.25930/) (2026)
- [Neural ODE enhanced linear mixed effect models for estimating complex association patterns of time-varying covariates with the marker trajectory](https://mlsanity.com/paper/2608.29714/) (2026)

## Abstract (arXiv)

Recent advances to combine structured regression models and deep neural networks for better interpretability, more expressiveness, and statistically valid uncertainty quantification demonstrate the versatility of semi-structured neural networks (SSNs). We show that techniques to properly identify the contributions of the different model components in SSNs, however, lead to suboptimal network estimation, slower convergence, and degenerated or erroneous predictions. In order to solve these problems while preserving favorable model properties, we propose a non-invasive post-hoc orthogonalization (PHO) that guarantees identifiability of model components and provides better estimation and prediction quality. Our theoretical findings are supported by numerical experiments, a benchmark comparison as well as a real-world application to COVID-19 infections.