# Adapting Pre-Trained Vision Models for Novel Instance Detection and Segmentation

arXiv: https://arxiv.org/abs/2405.17859 · HTML: https://mlsanity.com/paper/2405.17859/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 85 of its 105 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 30 of 85)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)

## Built on by (most cited first, 17 of 17)

- [BOP Challenge 2024 on Model-Based and Model-Free 6D Object Pose Estimation](https://mlsanity.com/paper/2504.02812/) (2025)
- [Accurate and efficient zero-shot 6D pose estimation with frozen foundation models](https://mlsanity.com/paper/2506.09784/) (2025)
- [6D Object Pose Tracking in Internet Videos for Robotic Manipulation](https://mlsanity.com/paper/2503.10307/) (2025)
- [XYZ-IBD: Benchmarking Robust 6D Object Pose Estimation under Real-World Industrial Complexity](https://mlsanity.com/paper/2506.00599/) (2025)
- [Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2508.04122/) (2025)
- [Multimodal Reference Visual Grounding](https://mlsanity.com/paper/2504.02876/) (2025)
- [MUSE: Model-based Uncertainty-aware Similarity Estimation for zero-shot 2D Object Detection and Segmentation](https://mlsanity.com/paper/2510.17866/) (2025)
- [NOCTIS: Novel Object Cyclic Threshold based Instance Segmentation](https://mlsanity.com/paper/2507.01463/) (2025)
- [OPFormer: Object Pose Estimation leveraging foundation model with geometric encoding](https://mlsanity.com/paper/2511.12614/) (2025)
- [Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perception](https://mlsanity.com/paper/2602.04343/) (2026)
- [AlignPose: Generalizable 6D Pose Estimation via Multi-view Feature-metric Alignment](https://mlsanity.com/paper/2512.20538/) (2025)
- [From Local Matches to Global Masks: Template-Guided Instance Detection and Segmentation in Open-World Scenes](https://mlsanity.com/paper/2603.03577/) (2026)
- [Decoupled Prototype Matching with Vision Foundation Models for Few-Shot Industrial Object Detection](https://mlsanity.com/paper/2604.26404/) (2026)
- [SOS! : A Streamlined Object-Conditional Transformer for Model-free Segmentation](https://mlsanity.com/paper/2608.15295/) (2026)
- [CLON: Cue-Calibrated Linguistic Object Onboarding for Zero-Shot 6D Pose Front-Ends](https://mlsanity.com/paper/2609.04784/) (2026)
- [Task-driven Processing with Coarse-to-Fine Glimpse-based Active Perception](https://mlsanity.com/paper/2609.09025/) (2026)
- [WAPR: A Foundation Model for Wide-Angle Refinement in Unseen Object Pose Estimation](https://mlsanity.com/paper/2610.09535/) (2026)

## Abstract (arXiv)

Novel Instance Detection and Segmentation (NIDS) aims at detecting and segmenting novel object instances given a few examples of each instance. We propose a unified, simple, yet effective framework (NIDS-Net) comprising object proposal generation, embedding creation for both instance templates and proposal regions, and embedding matching for instance label assignment. Leveraging recent advancements in large vision methods, we utilize Grounding DINO and Segment Anything Model (SAM) to obtain object proposals with accurate bounding boxes and masks. Central to our approach is the generation of high-quality instance embeddings. We utilized foreground feature averages of patch embeddings from the DINOv2 ViT backbone, followed by refinement through a weight adapter mechanism that we introduce. We show experimentally that our weight adapter can adjust the embeddings locally within their feature space and effectively limit overfitting in the few-shot setting. Furthermore, the weight adapter optimizes weights to enhance the distinctiveness of instance embeddings during similarity computation. This methodology enables a straightforward matching strategy that results in significant performance gains. Our framework surpasses current state-of-the-art methods, demonstrating notable improvements in four detection datasets. In the segmentation tasks on seven core datasets of the BOP challenge, our method outperforms the leading published RGB methods and remains competitive with the best RGB-D method. We have also verified our method using real-world images from a Fetch robot and a RealSense camera. Project Page: https://irvlutd.github.io/NIDSNet/