# Discrete Tokenization Unlocks Transformers for Calibrated Tabular Forecasting

arXiv: https://arxiv.org/abs/2603.07448 · HTML: https://mlsanity.com/paper/2603.07448/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 8 Mar 2026.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [DeepSurv: Personalized Treatment Recommender System Using A Cox Proportional Hazards Deep Neural Network](https://mlsanity.com/paper/1606.00931/) (2016)
- [Transformer Hawkes Process](https://mlsanity.com/paper/2002.09291/) (2020)
- [Chronos-2: From Univariate to Universal Forecasting](https://mlsanity.com/paper/2510.15821/) (2025)
- [Intensity-Free Learning of Temporal Point Processes](https://mlsanity.com/paper/1909.12127/) (2019)
- [The Effectiveness of Discretization in Forecasting: An Empirical Study on Neural Time Series Models](https://mlsanity.com/paper/2005.10111/) (2020)

## Abstract (arXiv)

Gradient boosting still dominates Transformers on tabular benchmarks. Our tokenizer uses a deliberately simplistic discretized vocabulary so we can highlight how even basic tokenization unlocks the power of attention on tabular features, yet it already outperforms tuned gradient boosting when combined with Gaussian smoothing. Our solution discretizes environmental context while smoothing labels with adaptive Gaussians, yielding calibrated PDFs. On 600K entities (5M training examples) we outperform tuned XGBoost by 10.8% (35.94s vs 40.31s median MAE) and achieve KS=0.0045 with the adaptive-sigma checkpoint selected to minimize KS rather than median MAE. Ablations confirm architecture matters: losing sequential ordering costs about 2.0%, dropping the time-delta tokens costs about 1.8%, and a stratified calibration analysis reveals where miscalibration persists.