# Spatial Memory for Context Reasoning in Object Detection

arXiv: https://arxiv.org/abs/1704.04224 · HTML: https://mlsanity.com/paper/1704.04224/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2017.

## Built on by (most cited first, 22 of 22)

- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [Progressive End-to-End Object Detection in Crowded Scenes](https://mlsanity.com/paper/2203.07669/) (2022)
- [Object Detection with Multimodal Large Vision-Language Models: An In-depth Review](https://mlsanity.com/paper/2508.19294/) (2025)
- [ARM3D: Attention-based relation module for indoor 3D object detection](https://mlsanity.com/paper/2202.09715/) (2022)
- [IDRNet: Intervention-Driven Relation Network for Semantic Segmentation](https://mlsanity.com/paper/2310.10755/) (2023)
- [Graphonomy: Universal Image Parsing via Graph Reasoning and Transfer](https://mlsanity.com/paper/2101.10620/) (2021)
- [Transformer-based Context Condensation for Boosting Feature Pyramids in Object Detection](https://mlsanity.com/paper/2207.06603/) (2022)
- [In Defense of Clip-based Video Relation Detection](https://mlsanity.com/paper/2307.08984/) (2023)
- [Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformer](https://mlsanity.com/paper/2407.08460/) (2024)
- [A Review of 3D Object Detection with Vision-Language Models](https://mlsanity.com/paper/2504.18738/) (2025)
- [A Survey on Deep Domain Adaptation and Tiny Object Detection Challenges, Techniques and Datasets](https://mlsanity.com/paper/2107.07927/) (2021)
- [The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering](https://mlsanity.com/paper/2501.07109/) (2025)
- [Visual-Linguistic Agent: Towards Collaborative Contextual Object Reasoning](https://mlsanity.com/paper/2411.10252/) (2024)
- [Low Light Image Enhancement via Global and Local Context Modeling](https://mlsanity.com/paper/2101.00850/) (2021)
- [3DRM:Pair-wise relation module for 3D object detection](https://mlsanity.com/paper/2202.09721/) (2022)
- [Analysis of Visual Reasoning on One-Stage Object Detection](https://mlsanity.com/paper/2202.13115/) (2022)
- [Exploring Contextual Relationships for Cervical Abnormal Cell Detection](https://mlsanity.com/paper/2207.04693/) (2022)
- [GLOW: Global Layout Aware Attacks on Object Detection](https://mlsanity.com/paper/2302.14166/) (2023)
- [Imagine the Unseen: Occluded Pedestrian Detection via Adversarial Feature Completion](https://mlsanity.com/paper/2405.01311/) (2024)
- [HR-RCNN: Hierarchical Relational Reasoning for Object Detection](https://mlsanity.com/paper/2110.13892/) (2021)
- [Small Object Detection Model with Spatial Laplacian Pyramid Attention and Multi-Scale Features Enhancement in Aerial Images](https://mlsanity.com/paper/2602.23031/) (2026)
- [Context-Aware Feature-Fusion for Co-occurring Object Detection in Autonomous Driving](https://mlsanity.com/paper/2606.12628/) (2026)

## Abstract (arXiv)

Modeling instance-level context and object-object relationships is extremely challenging. It requires reasoning about bounding boxes of different classes, locations etc. Above all, instance-level spatial reasoning inherently requires modeling conditional distributions on previous detections. Unfortunately, our current object detection systems do not have any bf memory to remember what to condition on! The state-of-the-art object detectors still detect all object in parallel followed by non-maximal suppression (NMS). While memory has been used for tasks such as captioning, they mostly use image-level memory cells without capturing the spatial layout. On the other hand, modeling object-object relationships requires bf spatial reasoning -- not only do we need a memory to store the spatial layout, but also a effective reasoning module to extract spatial patterns. This paper presents a conceptually simple yet powerful solution -- Spatial Memory Network (SMN), to model the instance-level context efficiently and effectively. Our spatial memory essentially assembles object instances back into a pseudo "image" representation that is easy to be fed into another ConvNet for object-object context reasoning. This leads to a new sequential reasoning architecture where image and memory are processed in parallel to obtain detections which update the memory again. We show our SMN direction is promising as it provides 2.2% improvement over baseline Faster RCNN on the COCO dataset so far.