# Marformer: A Transformer for Predicting Missing Data Distributions

arXiv: https://arxiv.org/abs/2610.12379 · HTML: https://mlsanity.com/paper/2610.12379/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 83 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [GAIN: Missing Data Imputation using Generative Adversarial Nets](https://mlsanity.com/paper/1806.02920/) (2018)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Handling Incomplete Heterogeneous Data using VAEs](https://mlsanity.com/paper/1807.03653/) (2018)
- [Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures](https://mlsanity.com/paper/1409.2574/) (2014)
- [MisGAN: Learning from Incomplete Data with Generative Adversarial Networks](https://mlsanity.com/paper/1902.09599/) (2019)
- [Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation](https://mlsanity.com/paper/2208.11646/) (2022)
- [not-MIWAE: Deep Generative Modelling with Missing not at Random Data](https://mlsanity.com/paper/2006.12871/) (2020)
- [PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences](https://mlsanity.com/paper/2406.08469/) (2024)
- [ReMasker: Imputing Tabular Data with Masked Autoencoding](https://mlsanity.com/paper/2309.13793/) (2023)
- [Active Evaluation Acquisition for Efficient LLM Benchmarking](https://mlsanity.com/paper/2410.05952/) (2024)
- [Reinforcement Learning with Efficient Active Feature Acquisition](https://mlsanity.com/paper/2011.00825/) (2020)
- [MIWAE: Deep Generative Modelling and Imputation of Incomplete Data](https://mlsanity.com/paper/1812.02633/) (2018)
- [Not Another Imputation Method: A Transformer-based Model for Missing Values in Tabular Datasets](https://mlsanity.com/paper/2407.11540/) (2024)
- [Unmasking Trees for Tabular Data](https://mlsanity.com/paper/2407.05593/) (2024)
- [Universal Marginaliser for Deep Amortised Inference for Probabilistic Programs](https://mlsanity.com/paper/1910.07474/) (2019)

## Abstract (arXiv)

Real decisions are made under incomplete information. If we observe only some of the random variables we need, we can predict the others. The conditional marginals over the missing variables are the key ingredient for computing Bayes risk and Value of Information (VOI), the expected gain from acquiring one more observation before deciding. We present the Marformer, a Transformer trained to directly predict conditional marginals given any set of observed values. Like BERT, which is trained to predict missing words from context, the Marformer constructs a hidden-vector representation for each distribution p(Xi) and iteratively refines it through attention to other distributions p(Xj). Unlike generative approaches, the Marformer does not model the full joint distribution, requires no domain knowledge of the data-generating process, and makes all predictions in a single forward pass. We evaluate across three synthetic domains with missing data---Bayesian networks, discretized multivariate Gaussians, and structured annotation data. The Marformer can match or outperform classical missing-data methods, even when those methods are given the true model family and prior that generated the synthetic data. We also evaluate on a real annotation dataset, where the Marformer outperforms the evaluated baselines at the largest training size. In both cases, the Marformer is substantially faster than the evaluated generative baselines.