# Memory-Based Neighbourhood Embedding for Visual Recognition

arXiv: https://arxiv.org/abs/1908.04992 · HTML: https://mlsanity.com/paper/1908.04992/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2019.

## Built on by (most cited first, 16 of 16)

- [UniHCP: A Unified Model for Human-Centric Perceptions](https://mlsanity.com/paper/2303.02936/) (2023)
- [Layerwise Optimization by Gradient Decomposition for Continual Learning](https://mlsanity.com/paper/2105.07561/) (2021)
- [Exploring Classification Equilibrium in Long-Tailed Object Detection](https://mlsanity.com/paper/2108.07507/) (2021)
- [Mining Contextual Information Beyond Image for Semantic Segmentation](https://mlsanity.com/paper/2108.11819/) (2021)
- [Complementary Relation Contrastive Distillation](https://mlsanity.com/paper/2103.16367/) (2021)
- [HumanBench: Towards General Human-centric Perception with Projector Assisted Pretraining](https://mlsanity.com/paper/2303.05675/) (2023)
- [Duplex Contextual Relation Network for Polyp Segmentation](https://mlsanity.com/paper/2103.06725/) (2021)
- [DATR: Unsupervised Domain Adaptive Detection Transformer with Dataset-Level Adaptation and Prototypical Alignment](https://mlsanity.com/paper/2405.11765/) (2024)
- [InsCLR: Improving Instance Retrieval with Self-Supervision](https://mlsanity.com/paper/2112.01390/) (2021)
- [MCIBI++: Soft Mining Contextual Information Beyond Image for Semantic Segmentation](https://mlsanity.com/paper/2209.04471/) (2022)
- [Revisiting Local Descriptor for Improved Few-Shot Classification](https://mlsanity.com/paper/2103.16009/) (2021)
- [Universal Cross-Domain Retrieval: Generalizing Across Classes and Domains](https://mlsanity.com/paper/2108.08356/) (2021)
- [DAS: Densely-Anchored Sampling for Deep Metric Learning](https://mlsanity.com/paper/2208.00119/) (2022)
- [Class-level Multiple Distributions Representation are Necessary for Semantic Segmentation](https://mlsanity.com/paper/2303.08029/) (2023)
- [Adaptive Cross Batch Normalization for Metric Learning](https://mlsanity.com/paper/2303.17127/) (2023)
- [Segmentation-Assisted Brain MRI Synthesis with Cross-Image Multi-Contrast Feature Memory Bank Retrieval Augmentation](https://mlsanity.com/paper/2606.08421/) (2026)

## Abstract (arXiv)

Learning discriminative image feature embeddings is of great importance to visual recognition. To achieve better feature embeddings, most current methods focus on designing different network structures or loss functions, and the estimated feature embeddings are usually only related to the input images. In this paper, we propose Memory-based Neighbourhood Embedding (MNE) to enhance a general CNN feature by considering its neighbourhood. The method aims to solve two critical problems, i.e., how to acquire more relevant neighbours in the network training and how to aggregate the neighbourhood information for a more discriminative embedding. We first augment an episodic memory module into the network, which can provide more relevant neighbours for both training and testing. Then the neighbours are organized in a tree graph with the target instance as the root node. The neighbourhood information is gradually aggregated to the root node in a bottom-up manner, and aggregation weights are supervised by the class relationships between the nodes. We apply MNE on image search and few shot learning tasks. Extensive ablation studies demonstrate the effectiveness of each component, and our method significantly outperforms the state-of-the-art approaches.