# Online Learning via Memory: Retrieval-Augmented Detector Adaptation

arXiv: https://arxiv.org/abs/2409.10716 · HTML: https://mlsanity.com/paper/2409.10716/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 14 of its 14 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [DOTA: A Large-scale Dataset for Object Detection in Aerial Images](https://mlsanity.com/paper/1711.10398/) (2017)
- [Retrieval Augmented Classification for Long-Tail Visual Recognition](https://mlsanity.com/paper/2202.11233/) (2022)
- [Learning Customized Visual Models with Retrieval-Augmented Knowledge](https://mlsanity.com/paper/2301.07094/) (2023)
- [Retrieval-Augmented Open-Vocabulary Object Detection](https://mlsanity.com/paper/2404.05687/) (2024)

## Abstract (arXiv)

This paper presents a novel way of online adapting any off-the-shelf object detection model to a novel domain without retraining the detector model. Inspired by how humans quickly learn knowledge of a new subject (e.g., memorization), we allow the detector to look up similar object concepts from memory during test time. This is achieved through a retrieval augmented classification (RAC) module together with a memory bank that can be flexibly updated with new domain knowledge. We experimented with various off-the-shelf open-set detector and close-set detectors. With only a tiny memory bank (e.g., 10 images per category) and being training-free, our online learning method could significantly outperform baselines in adapting a detector to novel domains.