# End-to-End Compression for Tabular Foundation Models

arXiv: https://arxiv.org/abs/2602.05649 · HTML: https://mlsanity.com/paper/2602.05649/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 5 Feb 2026.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [Amortized Inference for Causal Structure Learning](https://mlsanity.com/paper/2205.12934/) (2022)
- [Benchmarking Distribution Shift in Tabular Data with TableShift](https://mlsanity.com/paper/2312.07577/) (2023)
- [MATE: Multi-view Attention for Table Transformer Efficiency](https://mlsanity.com/paper/2109.04312/) (2021)
- [A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities](https://mlsanity.com/paper/2502.17361/) (2025)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [ConTextTab: A Semantics-Aware Tabular In-Context Learner](https://mlsanity.com/paper/2506.10707/) (2025)
- [Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data](https://mlsanity.com/paper/2411.10634/) (2024)
- [TabFlex: Scaling Tabular Learning to Millions with Linear Attention](https://mlsanity.com/paper/2506.05584/) (2025)
- [Transformers Meet Relational Databases](https://mlsanity.com/paper/2412.05218/) (2024)
- [Interpretable Mesomorphic Networks for Tabular Data](https://mlsanity.com/paper/2305.13072/) (2023)
- [TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments](https://mlsanity.com/paper/2501.18935/) (2025)

## Built on by (most cited first, 6 of 6)

- [LUCoS: Latent Unsupervised Context Selection for Tabular Foundation Models](https://mlsanity.com/paper/2605.27254/) (2026)
- [Memory Efficient Tabular Foundation Models](https://mlsanity.com/paper/2607.27546/) (2026)
- [GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models](https://mlsanity.com/paper/2608.18849/) (2026)
- [Reinforcement Learning Foundation Models Should Already Be A Thing](https://mlsanity.com/paper/2606.18812/) (2026)
- [Distillation of Tabular Foundation Models into Efficient Predictors](https://mlsanity.com/paper/2610.01435/) (2026)
- [Tabular Foundation Models Are Effectively Shallow](https://mlsanity.com/paper/2610.13253/) (2026)

## Abstract (arXiv)

The long-standing dominance of gradient-boosted decision trees for tabular data has recently been challenged by in-context learning tabular foundation models. In-context learning methods fit and predict in one forward pass without parameter updates by leveraging the training data as context for predicting on query test points. While recent tabular foundation models achieve state-of-the-art performance, their transformer architecture based on the attention mechanism has quadratic complexity regarding dataset size, which in turn increases the overhead on training and inference time, and limits the capacity of the models to handle large-scale datasets. In this work, we propose TACO, an end-to-end tabular compression model that compresses the training dataset in a latent space. We test our method on the TabArena benchmark, where our proposed method is up to 94x faster in inference time, while consuming up to 97% less memory compared to the state-of-the-art tabular transformer architecture, all while retaining performance without significant degradation. Lastly, our method not only scales better with increased dataset sizes, but it also achieves better performance compared to other baselines.