# SEMF: Supervised Expectation-Maximization Framework for Predicting Intervals

arXiv: https://arxiv.org/abs/2405.18176 · HTML: https://mlsanity.com/paper/2405.18176/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 69 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 23 of 23)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [Statistical guarantees for the EM algorithm: From population to sample-based analysis](https://mlsanity.com/paper/1408.2156/) (2014)
- [High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach](https://mlsanity.com/paper/1802.07167/) (2018)
- [Sequential Predictive Conformal Inference for Time Series](https://mlsanity.com/paper/2212.03463/) (2022)
- [Uncertainty in Gradient Boosting via Ensembles](https://mlsanity.com/paper/2006.10562/) (2020)
- [Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning](https://mlsanity.com/paper/2004.00163/) (2020)
- [Analyzing the Role of Model Uncertainty for Electronic Health Records](https://mlsanity.com/paper/1906.03842/) (2019)
- [Processing of missing data by neural networks](https://mlsanity.com/paper/1805.07405/) (2018)
- [Multi-Modal Anomaly Detection for Unstructured and Uncertain Environments](https://mlsanity.com/paper/2012.08637/) (2020)
- [On training locally adaptive CP](https://mlsanity.com/paper/2306.04648/) (2023)
- [Online Learning with Continuous Ranked Probability Score](https://mlsanity.com/paper/1902.10173/) (2019)
- [Semi-supervised Learning with the EM Algorithm: A Comparative Study between Unstructured and Structured Prediction](https://mlsanity.com/paper/2008.12442/) (2020)
- [UCSL : A Machine Learning Expectation-Maximization framework for Unsupervised Clustering driven by Supervised Learning](https://mlsanity.com/paper/2107.01988/) (2021)
- [Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift](https://mlsanity.com/paper/2405.10302/) (2024)
- [Expectation Maximization Pseudo Labels](https://mlsanity.com/paper/2305.01747/) (2023)

## Built on by (most cited first, 2 of 2)

- [CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk](https://mlsanity.com/paper/2507.08150/) (2025)
- [Conformal Calibration for Multi-Modal Regression with Missing Modalities](https://mlsanity.com/paper/2608.07795/) (2026)

## Abstract (arXiv)

This work introduces the Supervised Expectation-Maximization Framework (SEMF), a versatile and model-agnostic approach for generating prediction intervals with any ML model. SEMF extends the Expectation-Maximization algorithm, traditionally used in unsupervised learning, to a supervised context, leveraging latent variable modeling for uncertainty estimation. Through extensive empirical evaluation of diverse simulated distributions and 11 real-world tabular datasets, SEMF consistently produces narrower prediction intervals while maintaining the desired coverage probability, outperforming traditional quantile regression methods. Furthermore, without using the quantile (pinball) loss, SEMF allows point predictors, including gradient-boosted trees and neural networks, to be calibrated with conformal quantile regression. The results indicate that SEMF enhances uncertainty quantification under diverse data distributions and is particularly effective for models that otherwise struggle with inherent uncertainty representation.