# Integrating Random Effects in Deep Neural Networks

arXiv: https://arxiv.org/abs/2206.03314 · HTML: https://mlsanity.com/paper/2206.03314/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 45 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 12 of 12)

- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)](https://mlsanity.com/paper/1503.01057/) (2015)
- [Deep Learning for Spatio-Temporal Data Mining: A Survey](https://mlsanity.com/paper/1906.04928/) (2019)
- [Stochastic Variational Deep Kernel Learning](https://mlsanity.com/paper/1611.00336/) (2016)
- [Entity Embeddings of Categorical Variables](https://mlsanity.com/paper/1604.06737/) (2016)
- [Application of Deep Convolutional Neural Networks for Detecting Extreme Weather in Climate Datasets](https://mlsanity.com/paper/1605.01156/) (2016)
- [Multi-Source Social Feedback of Online News Feeds](https://mlsanity.com/paper/1801.07055/) (2018)
- [Airbnb Price Prediction Using Machine Learning and Sentiment Analysis](https://mlsanity.com/paper/1907.12665/) (2019)
- [Deep Learning for Real Time Crime Forecasting](https://mlsanity.com/paper/1707.03340/) (2017)
- [Learning Neural Networks with Adaptive Regularization](https://mlsanity.com/paper/1907.06288/) (2019)
- [Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits](https://mlsanity.com/paper/2111.10461/) (2021)

## Built on by (most cited first, 22 of 22)

- [Machine Learning with High-Cardinality Categorical Features in Actuarial Applications](https://mlsanity.com/paper/2301.12710/) (2023)
- [Neural Mixed Effects for Nonlinear Personalized Predictions](https://mlsanity.com/paper/2306.08149/) (2023)
- [Shortcut Learning in Binary Classifier Black Boxes: Applications to Voice Anti-Spoofing and Biometrics](https://mlsanity.com/paper/2601.17782/) (2026)
- [Enabling Mixed Effects Neural Networks for Diverse, Clustered Data Using Monte Carlo Methods](https://mlsanity.com/paper/2407.01115/) (2024)
- [A Comparison of Machine Learning Methods for Data with High-Cardinality Categorical Variables](https://mlsanity.com/paper/2307.02071/) (2023)
- [Subject-specific Deep Neural Networks for Count Data with High-cardinality Categorical Features](https://mlsanity.com/paper/2310.11654/) (2023)
- [Reducing the dimensionality and granularity in hierarchical categorical variables](https://mlsanity.com/paper/2403.03613/) (2024)
- [Hamiltonian Monte Carlo Inference of Marginalized Linear Mixed-Effects Models](https://mlsanity.com/paper/2410.24079/) (2024)
- [Integrating Random Effects in Variational Autoencoders for Dimensionality Reduction of Correlated Data](https://mlsanity.com/paper/2412.16899/) (2024)
- [Online Federation For Mixtures of Proprietary Agents with Black-Box Encoders](https://mlsanity.com/paper/2505.00216/) (2025)
- [Neural Generalized Mixed-Effects Models](https://mlsanity.com/paper/2604.10976/) (2026)
- [Probabilistic size-and-shape functional mixed models](https://mlsanity.com/paper/2411.18416/) (2024)
- [Scalable Krylov Subspace Methods for Generalized Mixed-Effects Models with Crossed Random Effects](https://mlsanity.com/paper/2505.09552/) (2025)
- [Vecchia-Inducing-Points Full-Scale Approximations for Gaussian Processes](https://mlsanity.com/paper/2507.05064/) (2025)
- [MMbeddings: Parameter-Efficient, Low-Overfitting Probabilistic Embeddings Inspired by Nonlinear Mixed Models](https://mlsanity.com/paper/2510.22198/) (2025)
- [The Climber's Grip -- Personalized Deep Learning Models for Fear and Muscle Activity in Climbing](https://mlsanity.com/paper/2603.26575/) (2026)
- [MixINN: Accelerating Plant Breeding by Combining Mixed Models and Deep Learning for Interaction Prediction](https://mlsanity.com/paper/2605.04744/) (2026)
- [Bayesian meta-learning for modeling Alzheimer's disease progression](https://mlsanity.com/paper/2606.02228/) (2026)
- [Integrating Neural Encoders in Bayesian Generalized Linear Mixed Models for Multimodal Data](https://mlsanity.com/paper/2607.04647/) (2026)
- [Structure Learning on Clustered Data](https://mlsanity.com/paper/2607.08238/) (2026)
- [Neural ODE enhanced linear mixed effect models for estimating complex association patterns of time-varying covariates with the marker trajectory](https://mlsanity.com/paper/2608.29714/) (2026)
- [NPBoost: Neural Processes with Gradient-Boosted Fixed Effects](https://mlsanity.com/paper/2609.28122/) (2026)

## Abstract (arXiv)

Modern approaches to supervised learning like deep neural networks (DNNs) typically implicitly assume that observed responses are statistically independent. In contrast, correlated data are prevalent in real-life large-scale applications, with typical sources of correlation including spatial, temporal and clustering structures. These correlations are either ignored by DNNs, or ad-hoc solutions are developed for specific use cases. We propose to use the mixed models framework to handle correlated data in DNNs. By treating the effects underlying the correlation structure as random effects, mixed models are able to avoid overfitted parameter estimates and ultimately yield better predictive performance. The key to combining mixed models and DNNs is using the Gaussian negative log-likelihood (NLL) as a natural loss function that is minimized with DNN machinery including stochastic gradient descent (SGD). Since NLL does not decompose like standard DNN loss functions, the use of SGD with NLL presents some theoretical and implementation challenges, which we address. Our approach which we call LMMNN is demonstrated to improve performance over natural competitors in various correlation scenarios on diverse simulated and real datasets. Our focus is on a regression setting and tabular datasets, but we also show some results for classification. Our code is available at https://github.com/gsimchoni/lmmnn.