# From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning

arXiv: https://arxiv.org/abs/2604.05635 · HTML: https://mlsanity.com/paper/2604.05635/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 61 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)
- [TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks](https://mlsanity.com/paper/2402.11137/) (2024)
- [Mambular: A Sequential Model for Tabular Deep Learning](https://mlsanity.com/paper/2408.06291/) (2024)
- [Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning](https://mlsanity.com/paper/2511.02818/) (2025)
- [A Survey on Deep Tabular Learning](https://mlsanity.com/paper/2410.12034/) (2024)
- [Arithmetic Feature Interaction Is Necessary for Deep Tabular Learning](https://mlsanity.com/paper/2402.02334/) (2024)
- [Structural Neural Additive Models: Enhanced Interpretable Machine Learning](https://mlsanity.com/paper/2302.09275/) (2023)
- [Interpretable Mesomorphic Networks for Tabular Data](https://mlsanity.com/paper/2305.13072/) (2023)
- [Free-Knots Kolmogorov-Arnold Network: On the Analysis of Spline Knots and Advancing Stability](https://mlsanity.com/paper/2501.09283/) (2025)
- [Function Basis Encoding of Numerical Features in Factorization Machines](https://mlsanity.com/paper/2305.14528/) (2023)

## Abstract (arXiv)

Numerical preprocessing remains a critical component of tabular deep learning, as the representation of continuous features can strongly affect downstream performance. We systematically study spline-based numerical encodings, including B-splines, M-splines, and integrated splines (I-splines), under uniform, quantile-based, target-aware, and learnable-knot placement. For the learnable variants, we adopt a differentiable knot parameterization that enables stable end-to-end optimization of knot locations jointly with the backbone. We evaluate these encodings on a diverse collection of public regression and classification datasets using MLP, ResNet, and FT-Transformer backbones, and compare them against common numerical preprocessing baselines. Our results show that the effectiveness of numerical encoding depends strongly on the task, encoding size, and backbone. For classification, piecewise-linear encoding (PLE) is the most robust choice overall, while spline-based encodings remain competitive. For regression, no single encoding dominates, with performance depending on the spline family and knot-placement strategy, and larger gains generally observed for MLP and ResNet than for FT-Transformer. Learnable-knot variants can be optimized stably but may substantially increase training cost. Overall, numerical encodings should therefore be assessed jointly in terms of predictive performance and computational overhead. The implementation is publicly available at https://github.com/mkumar73/tdl-numerical-encodings/.