# Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains

arXiv: https://arxiv.org/abs/2405.07414 · HTML: https://mlsanity.com/paper/2405.07414/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 13 May 2024.

## Builds on (most cited first, 28 of 28)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [TabDDPM: Modelling Tabular Data with Diffusion Models](https://mlsanity.com/paper/2209.15421/) (2022)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption](https://mlsanity.com/paper/2106.15147/) (2021)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning](https://mlsanity.com/paper/2110.04361/) (2021)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [Gradient Boosting Neural Networks: GrowNet](https://mlsanity.com/paper/2002.07971/) (2020)
- [Revisiting Pretraining Objectives for Tabular Deep Learning](https://mlsanity.com/paper/2207.03208/) (2022)
- [T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interaction](https://mlsanity.com/paper/2211.16887/) (2022)
- [Trompt: Towards a Better Deep Neural Network for Tabular Data](https://mlsanity.com/paper/2305.18446/) (2023)
- [HYTREL: Hypergraph-enhanced Tabular Data Representation Learning](https://mlsanity.com/paper/2307.08623/) (2023)
- [Regression as Classification: Influence of Task Formulation on Neural Network Features](https://mlsanity.com/paper/2211.05641/) (2022)
- [A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning](https://mlsanity.com/paper/2311.05877/) (2023)
- [Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning](https://mlsanity.com/paper/2212.08663/) (2022)

## Built on by (most cited first, 12 of 12)

- [Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzles](https://mlsanity.com/paper/2505.23590/) (2025)
- [TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks](https://mlsanity.com/paper/2406.19380/) (2024)
- [T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Data](https://mlsanity.com/paper/2410.05016/) (2024)
- [LLM Embeddings for Deep Learning on Tabular Data](https://mlsanity.com/paper/2502.11596/) (2025)
- [Position: Foundation Models for Tabular Data within Systemic Contexts Need Grounding](https://mlsanity.com/paper/2505.19825/) (2025)
- [EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognition](https://mlsanity.com/paper/2508.14130/) (2025)
- [Building Bridges between Regression, Clustering, and Classification](https://mlsanity.com/paper/2502.02996/) (2025)
- [MultiTab: A Comprehensive Benchmark Suite for Multi-Dimensional Evaluation in Tabular Domains](https://mlsanity.com/paper/2505.14312/) (2025)
- [Towards Interpretable Deep Neural Networks for Tabular Data](https://mlsanity.com/paper/2509.08617/) (2025)
- [XNNTab -- Interpretable Neural Networks for Tabular Data using Sparse Autoencoders](https://mlsanity.com/paper/2512.13442/) (2025)
- [When, Where, and How: Adaptive Binning for Tabular Self-Supervised Learning](https://mlsanity.com/paper/2606.19827/) (2026)
- [Constrained Tabular Diffusion for Finance](https://mlsanity.com/paper/2606.28674/) (2026)

## Abstract (arXiv)

The ability of deep networks to learn superior representations hinges on leveraging the proper inductive biases, considering the inherent properties of datasets. In tabular domains, it is critical to effectively handle heterogeneous features (both categorical and numerical) in a unified manner and to grasp irregular functions like piecewise constant functions. To address the challenges in the self-supervised learning framework, we propose a novel pretext task based on the classical binning method. The idea is straightforward: reconstructing the bin indices (either orders or classes) rather than the original values. This pretext task provides the encoder with an inductive bias to capture the irregular dependencies, mapping from continuous inputs to discretized bins, and mitigates the feature heterogeneity by setting all features to have category-type targets. Our empirical investigations ascertain several advantages of binning: capturing the irregular function, compatibility with encoder architecture and additional modifications, standardizing all features into equal sets, grouping similar values within a feature, and providing ordering information. Comprehensive evaluations across diverse tabular datasets corroborate that our method consistently improves tabular representation learning performance for a wide range of downstream tasks. The codes are available in https://github.com/kyungeun-lee/tabularbinning.