# Neural Nearest Neighbors Networks

arXiv: https://arxiv.org/abs/1810.12575 · HTML: https://mlsanity.com/paper/1810.12575/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2018.

## Built on by (most cited first, 30 of 98)

- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Taming 3DGS: High-Quality Radiance Fields with Limited Resources](https://mlsanity.com/paper/2406.15643/) (2024)
- [Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images](https://mlsanity.com/paper/2101.02824/) (2021)
- [Learning Enriched Features for Fast Image Restoration and Enhancement](https://mlsanity.com/paper/2205.01649/) (2022)
- [Practical Blind Image Denoising via Swin-Conv-UNet and Data Synthesis](https://mlsanity.com/paper/2203.13278/) (2022)
- [Towards Self-Explainable Graph Neural Network](https://mlsanity.com/paper/2108.12055/) (2021)
- [Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --](https://mlsanity.com/paper/2301.03362/) (2023)
- [HDRUNet: Single Image HDR Reconstruction with Denoising and Dequantization](https://mlsanity.com/paper/2105.13084/) (2021)
- [Masked Image Training for Generalizable Deep Image Denoising](https://mlsanity.com/paper/2303.13132/) (2023)
- [Dynamic Attentive Graph Learning for Image Restoration](https://mlsanity.com/paper/2109.06620/) (2021)
- [Differentiable Subset Pruning of Transformer Heads](https://mlsanity.com/paper/2108.04657/) (2021)
- [Rethinking Deep Image Prior for Denoising](https://mlsanity.com/paper/2108.12841/) (2021)
- [SIMPLE: A Gradient Estimator for k-Subset Sampling](https://mlsanity.com/paper/2210.01941/) (2022)
- [KBNet: Kernel Basis Network for Image Restoration](https://mlsanity.com/paper/2303.02881/) (2023)
- [ReconfigISP: Reconfigurable Camera Image Processing Pipeline](https://mlsanity.com/paper/2109.04760/) (2021)
- [Unsupervised Image Denoising in Real-World Scenarios via Self-Collaboration Parallel Generative Adversarial Branches](https://mlsanity.com/paper/2308.06776/) (2023)
- [Learning to Intervene on Concept Bottlenecks](https://mlsanity.com/paper/2308.13453/) (2023)
- [RetrievalFuse: Neural 3D Scene Reconstruction with a Database](https://mlsanity.com/paper/2104.00024/) (2021)
- [FBI-Denoiser: Fast Blind Image Denoiser for Poisson-Gaussian Noise](https://mlsanity.com/paper/2105.10967/) (2021)
- [Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers](https://mlsanity.com/paper/2412.10569/) (2024)
- [Content-Adaptive Non-Local Convolution for Remote Sensing Pansharpening](https://mlsanity.com/paper/2404.07543/) (2024)
- [Dead Pixel Test Using Effective Receptive Field](https://mlsanity.com/paper/2108.13576/) (2021)
- [Multi-Scale Adaptive Network for Single Image Denoising](https://mlsanity.com/paper/2203.04313/) (2022)
- [COLA-Net: Collaborative Attention Network for Image Restoration](https://mlsanity.com/paper/2103.05961/) (2021)
- [Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?](https://mlsanity.com/paper/2309.06891/) (2023)
- [All the attention you need: Global-local, spatial-channel attention for image retrieval](https://mlsanity.com/paper/2107.08000/) (2021)
- [DetarNet: Decoupling Translation and Rotation by Siamese Network for Point Cloud Registration](https://mlsanity.com/paper/2112.14059/) (2021)
- [Machine learning for modelling unstructured grid data in computational physics: a review](https://mlsanity.com/paper/2502.09346/) (2025)
- [Calibration of Machine Reading Systems at Scale](https://mlsanity.com/paper/2203.10623/) (2022)
- [LoMOE: Localized Multi-Object Editing via Multi-Diffusion](https://mlsanity.com/paper/2403.00437/) (2024)

## Abstract (arXiv)

Non-local methods exploiting the self-similarity of natural signals have been well studied, for example in image analysis and restoration. Existing approaches, however, rely on k-nearest neighbors (KNN) matching in a fixed feature space. The main hurdle in optimizing this feature space w.r.t. application performance is the non-differentiability of the KNN selection rule. To overcome this, we propose a continuous deterministic relaxation of KNN selection that maintains differentiability w.r.t. pairwise distances, but retains the original KNN as the limit of a temperature parameter approaching zero. To exploit our relaxation, we propose the neural nearest neighbors block (N3 block), a novel non-local processing layer that leverages the principle of self-similarity and can be used as building block in modern neural network architectures. We show its effectiveness for the set reasoning task of correspondence classification as well as for image restoration, including image denoising and single image super-resolution, where we outperform strong convolutional neural network (CNN) baselines and recent non-local models that rely on KNN selection in hand-chosen features spaces.