# DNNR: Differential Nearest Neighbors Regression

arXiv: https://arxiv.org/abs/2205.08434 · HTML: https://mlsanity.com/paper/2205.08434/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 44 references that are in the map.
- First posted on arXiv 17 May 2022.

## Builds on (most cited first, 12 of 12)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [CatBoost: gradient boosting with categorical features support](https://mlsanity.com/paper/1810.11363/) (2018)
- [A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization](https://mlsanity.com/paper/1905.01332/) (2019)
- [Rates of Convergence for Nearest Neighbor Classification](https://mlsanity.com/paper/1407.0067/) (2014)
- [DeepFit: 3D Surface Fitting via Neural Network Weighted Least Squares](https://mlsanity.com/paper/2003.10826/) (2020)
- [Deep Gaussian Processes for Regression using Approximate Expectation Propagation](https://mlsanity.com/paper/1602.04133/) (2016)
- [An adaptive nearest neighbor rule for classification](https://mlsanity.com/paper/1905.12717/) (2019)
- [Nearest Neighbour Based Estimates of Gradients: Sharp Nonasymptotic Bounds and Applications](https://mlsanity.com/paper/2006.15043/) (2020)

## Built on by (most cited first, 11 of 11)

- [TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://mlsanity.com/paper/2307.14338/) (2023)
- [Retrieval & Fine-Tuning for In-Context Tabular Models](https://mlsanity.com/paper/2406.05207/) (2024)
- [TALENT: A Tabular Analytics and Learning Toolbox](https://mlsanity.com/paper/2407.04057/) (2024)
- [A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities](https://mlsanity.com/paper/2502.17361/) (2025)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [Elliptical Attention](https://mlsanity.com/paper/2406.13770/) (2024)
- [Retrieval Augmented Time Series Forecasting](https://mlsanity.com/paper/2505.04163/) (2025)
- [TMPNN: High-Order Polynomial Regression Based on Taylor Map Factorization](https://mlsanity.com/paper/2307.16105/) (2023)
- [SoftStep: Learning Sparse Similarity Powers Deep Neighbor-Based Regression](https://mlsanity.com/paper/2506.08139/) (2025)
- [TabNSM: Neural Sparse Mixer for Tabular Regression](https://mlsanity.com/paper/2608.18026/) (2026)
- [Xiaomi-TabLDM: A Tabular Foundation Model Technical Report](https://mlsanity.com/paper/2609.03880/) (2026)

## Abstract (arXiv)

K-nearest neighbors (KNN) is one of the earliest and most established algorithms in machine learning. For regression tasks, KNN averages the targets within a neighborhood which poses a number of challenges: the neighborhood definition is crucial for the predictive performance as neighbors might be selected based on uninformative features, and averaging does not account for how the function changes locally. We propose a novel method called Differential Nearest Neighbors Regression (DNNR) that addresses both issues simultaneously: during training, DNNR estimates local gradients to scale the features; during inference, it performs an n-th order Taylor approximation using estimated gradients. In a large-scale evaluation on over 250 datasets, we find that DNNR performs comparably to state-of-the-art gradient boosting methods and MLPs while maintaining the simplicity and transparency of KNN. This allows us to derive theoretical error bounds and inspect failures. In times that call for transparency of ML models, DNNR provides a good balance between performance and interpretability.