# 3D Semantic MapNet: Building Maps for Multi-Object Re-Identification in 3D

arXiv: https://arxiv.org/abs/2403.13190 · HTML: https://mlsanity.com/paper/2403.13190/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 27 of 27)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Deep Learning for Person Re-identification: A Survey and Outlook](https://mlsanity.com/paper/2001.04193/) (2020)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019)
- [Rearrangement: A Challenge for Embodied AI](https://mlsanity.com/paper/2011.01975/) (2020)
- [Probabilistic 3D Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2001.05673/) (2020)
- [A Dataset for Developing and Benchmarking Active Vision](https://mlsanity.com/paper/1702.08272/) (2017)
- [Semantic Curiosity for Active Visual Learning](https://mlsanity.com/paper/2006.09367/) (2020)
- [On the Unreasonable Effectiveness of Centroids in Image Retrieval](https://mlsanity.com/paper/2104.13643/) (2021)
- [Object Rearrangement Using Learned Implicit Collision Functions](https://mlsanity.com/paper/2011.10726/) (2020)
- [VehicleNet: Learning Robust Visual Representation for Vehicle Re-identification](https://mlsanity.com/paper/2004.06305/) (2020)
- [ODAM: Object Detection, Association, and Mapping using Posed RGB Video](https://mlsanity.com/paper/2108.10165/) (2021)
- [Person Re-Identification with a Locally Aware Transformer](https://mlsanity.com/paper/2106.03720/) (2021)
- [FlipReID: Closing the Gap between Training and Inference in Person Re-Identification](https://mlsanity.com/paper/2105.05639/) (2021)
- [A Strong Baseline for Vehicle Re-Identification](https://mlsanity.com/paper/2104.10850/) (2021)
- [The Robotic Vision Scene Understanding Challenge](https://mlsanity.com/paper/2009.05246/) (2020)

## Built on by (most cited first, 1 of 1)

- [Lookalike3D: Seeing Double in 3D](https://mlsanity.com/paper/2603.24713/) (2026)

## Abstract (arXiv)

We study the task of 3D multi-object re-identification from embodied tours. Specifically, an agent is given two tours of an environment (e.g. an apartment) under two different layouts (e.g. arrangements of furniture). Its task is to detect and re-identify objects in 3D - e.g. a "sofa" moved from location A to B, a new "chair" in the second layout at location C, or a "lamp" from location D in the first layout missing in the second. To support this task, we create an automated infrastructure to generate paired egocentric tours of initial/modified layouts in the Habitat simulator using Matterport3D scenes, YCB and Google-scanned objects. We present 3D Semantic MapNet (3D-SMNet) - a two-stage re-identification model consisting of (1) a 3D object detector that operates on RGB-D videos with known pose, and (2) a differentiable object matching module that solves correspondence estimation between two sets of 3D bounding boxes. Overall, 3D-SMNet builds object-based maps of each layout and then uses a differentiable matcher to re-identify objects across the tours. After training 3D-SMNet on our generated episodes, we demonstrate zero-shot transfer to real-world rearrangement scenarios by instantiating our task in Replica, Active Vision, and RIO environments depicting rearrangements. On all datasets, we find 3D-SMNet outperforms competitive baselines. Further, we show jointly training on real and generated episodes can lead to significant improvements over training on real data alone.