# Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Later

arXiv: https://arxiv.org/abs/2407.03257 · HTML: https://mlsanity.com/paper/2407.03257/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 86 references that are in the map.
- First posted on arXiv 3 Jul 2024.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption](https://mlsanity.com/paper/2106.15147/) (2021)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning](https://mlsanity.com/paper/2110.04361/) (2021)
- [Analyzing and Improving Representations with the Soft Nearest Neighbor Loss](https://mlsanity.com/paper/1902.01889/) (2019)
- [NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning](https://mlsanity.com/paper/2106.01613/) (2021)
- [TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://mlsanity.com/paper/2307.14338/) (2023)

## Built on by (most cited first, 30 of 30)

- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [Griffin: Towards a Graph-Centric Relational Database Foundation Model](https://mlsanity.com/paper/2505.05568/) (2025)
- [A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities](https://mlsanity.com/paper/2502.17361/) (2025)
- [TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems](https://mlsanity.com/paper/2502.02527/) (2025)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [Multimodal Tabular Reasoning with Privileged Structured Information](https://mlsanity.com/paper/2506.04088/) (2025)
- [Relational In-Context Learning via Synthetic Pre-training with Structural Prior](https://mlsanity.com/paper/2603.03805/) (2026)
- [LLM Meeting Decision Trees on Tabular Data](https://mlsanity.com/paper/2505.17918/) (2025)
- [TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention](https://mlsanity.com/paper/2606.07345/) (2026)
- [LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence](https://mlsanity.com/paper/2609.17488/) (2026)
- [TabGemma: Text-Based Tabular ICL via LLM using Continued Pretraining and Retrieval](https://mlsanity.com/paper/2511.03570/) (2025)
- [Benchmarking Optimizers for MLPs in Tabular Deep Learning](https://mlsanity.com/paper/2604.15297/) (2026)
- [Modern Neural Networks for Small Tabular Datasets: The New Default for Field-Scale Digital Soil Mapping?](https://mlsanity.com/paper/2508.09888/) (2025)
- [Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraints](https://mlsanity.com/paper/2604.01841/) (2026)
- [EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Networks](https://mlsanity.com/paper/2502.06684/) (2025)
- [Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard](https://mlsanity.com/paper/2505.13421/) (2025)
- [Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning](https://mlsanity.com/paper/2603.24004/) (2026)
- [CalArena: A Large-Scale Post-Hoc Calibration Benchmark](https://mlsanity.com/paper/2605.30188/) (2026)
- [TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks](https://mlsanity.com/paper/2606.02384/) (2026)
- [Feature-aware Modulation for Learning from Temporal Tabular Data](https://mlsanity.com/paper/2512.03678/) (2025)
- [RamanPFN: learning from Raman spectral structure with a tabular foundation model](https://mlsanity.com/paper/2608.02157/) (2026)
- [Learning based on neurovectors for tabular data: a new neural network approach](https://mlsanity.com/paper/2506.07185/) (2025)
- [TFM-Retouche: A Lightweight Input-Space Adapter for Tabular Foundation Models](https://mlsanity.com/paper/2605.06047/) (2026)
- [Cross-Domain Feature Expansion for Tabular Medical Data via Knowledge Graphs Injection](https://mlsanity.com/paper/2606.31171/) (2026)
- [Hyperbolic Manifold Constrained Tabular Neural Network](https://mlsanity.com/paper/2607.09710/) (2026)
- [Xiaomi-TabLDM: A Tabular Foundation Model Technical Report](https://mlsanity.com/paper/2609.03880/) (2026)
- [Agentic Search Spaces for Tabular Machine Learning](https://mlsanity.com/paper/2609.16309/) (2026)
- [Explainable Predictive Condition-based Maintenance of Naval-Propulsion Systems using Fuzzy Logic](https://mlsanity.com/paper/2609.24250/) (2026)
- [TILE: Bridging LLMs and Table Foundation Models for Tabular Learning with Strings](https://mlsanity.com/paper/2610.17065/) (2026)

## Abstract (arXiv)

The widespread enthusiasm for deep learning has recently expanded into the domain of tabular data. Recognizing that the advancement in deep tabular methods is often inspired by classical methods, e.g., integration of nearest neighbors into neural networks, we investigate whether these classical methods can be revitalized with modern techniques. We revisit a differentiable version of K-nearest neighbors (KNN) -- Neighbourhood Components Analysis (NCA) -- originally designed to learn a linear projection to capture semantic similarities between instances, and seek to gradually add modern deep learning techniques on top. Surprisingly, our implementation of NCA using SGD and without dimensionality reduction already achieves decent performance on tabular data, in contrast to the results of using existing toolboxes like scikit-learn. Further equipping NCA with deep representations and additional training stochasticity significantly enhances its capability, being on par with the leading tree-based method CatBoost and outperforming existing deep tabular models in both classification and regression tasks on 300 datasets. We conclude our paper by analyzing the factors behind these improvements, including loss functions, prediction strategies, and deep architectures. The code is available at https://github.com/qile2000/LAMDA-TALENT.