# MatrixFormer: A Foundation Model for Matrix Completion

arXiv: https://arxiv.org/abs/2610.06751 · HTML: https://mlsanity.com/paper/2610.06751/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 49 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 9 of 9)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [Submodular Benchmark Selection](https://mlsanity.com/paper/2605.02209/) (2026)
- [CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation](https://mlsanity.com/paper/2506.02306/) (2025)
- [One Pipeline, Many Transformers: Pattern-Specific Imputation Specialists for Tabular Missing Data](https://mlsanity.com/paper/2510.02625/) (2025)

## Abstract (arXiv)

Matrix completion underlies problems from tabular imputation to causal inference, yet existing tabular foundation models treat it as entry-by-entry prediction, repeating context for every target and discarding the matrix's two-dimensional structure. We introduce MatrixFormer, a pre-trained matrix-native transformer that predicts a full distribution for every missing entry in a single forward pass. MatrixFormer is trained entirely on synthetic low-rank and latent-factor matrices under diverse missingness patterns. Applied zero-shot and with the same model weights, MatrixFormer achieves competitive performance on causal inference panel-data tasks, language-model benchmark-score completion, tabular imputation, and recommendation systems matrix completion. These results position MatrixFormer as a general-purpose foundation model for matrix completion.