# Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings

arXiv: https://arxiv.org/abs/2511.06961 · HTML: https://mlsanity.com/paper/2511.06961/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Builds on (most cited first, 21 of 21)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning](https://mlsanity.com/paper/2110.04361/) (2021)
- [Locally Sparse Neural Networks for Tabular Biomedical Data](https://mlsanity.com/paper/2106.06468/) (2021)
- [Interpretable Deep Clustering for Tabular Data](https://mlsanity.com/paper/2306.04785/) (2023)
- [SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training](https://mlsanity.com/paper/2505.24749/) (2025)
- [AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning](https://mlsanity.com/paper/2410.17881/) (2024)
- [Deep Unsupervised Feature Selection by Discarding Nuisance and Correlated Features](https://mlsanity.com/paper/2110.05306/) (2021)
- [Support Recovery with Stochastic Gates: Theory and Application for Linear Models](https://mlsanity.com/paper/2110.15960/) (2021)
- [Anomaly Detection with Variance Stabilized Density Estimation](https://mlsanity.com/paper/2306.00582/) (2023)
- [Contextual Feature Selection with Conditional Stochastic Gates](https://mlsanity.com/paper/2312.14254/) (2023)
- [Autoencoder Trees](https://mlsanity.com/paper/1409.7461/) (2014)
- [To Predict or Not To Predict? Proportionally Masked Autoencoders for Tabular Data Imputation](https://mlsanity.com/paper/2412.19152/) (2024)
- [FineGates: LLMs Finetuning with Compression using Stochastic Gates](https://mlsanity.com/paper/2412.12951/) (2024)

## Built on by (most cited first, 5 of 5)

- [Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity](https://mlsanity.com/paper/2602.09169/) (2026)
- [ZAYAN: Disentangled Contrastive Transformer for Tabular Remote Sensing Data](https://mlsanity.com/paper/2604.27606/) (2026)
- [Uncovering a Winning Lottery Ticket with Continuously Relaxed Bernoulli Gates](https://mlsanity.com/paper/2603.08914/) (2026)
- [Learning Permutation from Structure Without Supervision](https://mlsanity.com/paper/2605.25551/) (2026)
- [BSTabDiff: Block-Subunit Diffusion Priors for High-Dimensional Tabular Data Generation](https://mlsanity.com/paper/2606.09257/) (2026)

## Abstract (arXiv)

Deep neural networks often under-perform on tabular data due to their sensitivity to irrelevant features and a spectral bias toward smooth, low-frequency functions. These limitations hinder their ability to capture the sharp, high-frequency signals that often define tabular structure, especially under limited labeled samples. While self-supervised learning (SSL) offers promise in such settings, it remains challenging in tabular domains due to the lack of effective data augmentations. We propose a hybrid autoencoder that combines a neural encoder with an oblivious soft decision tree (OSDT) encoder, each guided by its own stochastic gating network that performs sample-specific feature selection. Together, these structurally different encoders and model-specific gating networks implement model-based augmentation, producing complementary input views tailored to each architecture. The two encoders, trained with a shared decoder and cross-reconstruction loss, learn distinct yet aligned representations that reflect their respective inductive biases. During training, the OSDT encoder (robust to noise and effective at modeling localized, high-frequency structure) guides the neural encoder toward representations more aligned with tabular data. At inference, only the neural encoder is used, preserving flexibility and SSL compatibility. Spectral analysis highlights the distinct inductive biases of each encoder. Our method achieves consistent gains in low-label classification and regression across diverse tabular datasets, outperforming deep and tree-based supervised baselines.