# SFD2: Semantic-guided Feature Detection and Description

arXiv: https://arxiv.org/abs/2304.14845 · HTML: https://mlsanity.com/paper/2304.14845/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 88 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 30 of 59)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [D2-Net: A Trainable CNN for Joint Detection and Description of Local Features](https://mlsanity.com/paper/1905.03561/) (2019)
- [DISK: Learning local features with policy gradient](https://mlsanity.com/paper/2006.13566/) (2020)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1606.02147/) (2016)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Working hard to know your neighbor's margins: Local descriptor learning loss](https://mlsanity.com/paper/1705.10872/) (2017)
- [R2D2: Repeatable and Reliable Detector and Descriptor](https://mlsanity.com/paper/1906.06195/) (2019)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [Understanding the Limitations of CNN-based Absolute Camera Pose Regression](https://mlsanity.com/paper/1903.07504/) (2019)
- [ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer](https://mlsanity.com/paper/2208.14201/) (2022)
- [LF-Net: Learning Local Features from Images](https://mlsanity.com/paper/1805.09662/) (2018)
- [Learning Less is More - 6D Camera Localization via 3D Surface Regression](https://mlsanity.com/paper/1711.10228/) (2017)
- [SOSNet: Second Order Similarity Regularization for Local Descriptor Learning](https://mlsanity.com/paper/1904.05019/) (2019)
- [ASLFeat: Learning Local Features of Accurate Shape and Localization](https://mlsanity.com/paper/2003.10071/) (2020)
- [Patch2Pix: Epipolar-Guided Pixel-Level Correspondences](https://mlsanity.com/paper/2012.01909/) (2020)
- [Learning Accurate Dense Correspondences and When to Trust Them](https://mlsanity.com/paper/2101.01710/) (2021)

## Built on by (most cited first, 28 of 28)

- [OmniGlue: Generalizable Feature Matching with Foundation Model Guidance](https://mlsanity.com/paper/2405.12979/) (2024)
- [Local Feature Matching Using Deep Learning: A Survey](https://mlsanity.com/paper/2401.17592/) (2024)
- [PatchDPO: Patch-level DPO for Finetuning-free Personalized Image Generation](https://mlsanity.com/paper/2412.03177/) (2024)
- [Segment Anything Model is a Good Teacher for Local Feature Learning](https://mlsanity.com/paper/2309.16992/) (2023)
- [MATCHA:Towards Matching Anything](https://mlsanity.com/paper/2501.14945/) (2025)
- [CoMatch: Dynamic Covisibility-Aware Transformer for Bilateral Subpixel-Level Semi-Dense Image Matching](https://mlsanity.com/paper/2503.23925/) (2025)
- [Learning to Produce Semi-dense Correspondences for Visual Localization](https://mlsanity.com/paper/2402.08359/) (2024)
- [Deep Learning Reforms Image Matching: A Survey and Outlook](https://mlsanity.com/paper/2506.04619/) (2025)
- [SRPose: Two-view Relative Pose Estimation with Sparse Keypoints](https://mlsanity.com/paper/2407.08199/) (2024)
- [Leveraging Semantic Cues from Foundation Vision Models for Enhanced Local Feature Correspondence](https://mlsanity.com/paper/2410.09533/) (2024)
- [Keypoint Semantic Integration for Improved Feature Matching in Outdoor Agricultural Environments](https://mlsanity.com/paper/2503.08843/) (2025)
- [DistillMatch: Leveraging Knowledge Distillation from Vision Foundation Model for Multimodal Image Matching](https://mlsanity.com/paper/2509.16017/) (2025)
- [Reprojection Errors as Prompts for Efficient Scene Coordinate Regression](https://mlsanity.com/paper/2409.04178/) (2024)
- [StructVPR++: Distill Structural and Semantic Knowledge with Weighting Samples for Visual Place Recognition](https://mlsanity.com/paper/2503.06601/) (2025)
- [Object-aware graph matching network for cross-domain remote sensing image localization](https://mlsanity.com/paper/2511.02489/) (2025)
- [Region Matters: Efficient and Reliable Region-Aware Visual Place Recognition](https://mlsanity.com/paper/2604.22390/) (2026)
- [Weatherproofing Retrieval for Localization with Generative AI and Geometric Consistency](https://mlsanity.com/paper/2402.09237/) (2024)
- [VRS-NeRF: Visual Relocalization with Sparse Neural Radiance Field](https://mlsanity.com/paper/2404.09271/) (2024)
- [GESS: Multi-cue Guided Local Feature Learning via Geometric and Semantic Synergy](https://mlsanity.com/paper/2604.05359/) (2026)
- [Matching Semantically Similar Non-Identical Objects](https://mlsanity.com/paper/2403.08227/) (2024)
- [PRAM: Place Recognition Anywhere Model for Efficient Visual Localization](https://mlsanity.com/paper/2404.07785/) (2024)
- [Map-Free Visual Relocalization Enhanced by Instance Knowledge and Depth Knowledge](https://mlsanity.com/paper/2408.13085/) (2024)
- [GMatch: A Lightweight, Geometry-Constrained Keypoint Matcher for Zero-Shot 6DoF Pose Estimation in Robotic Grasp Tasks](https://mlsanity.com/paper/2505.16144/) (2025)
- [Semantic and Feature Guided Uncertainty Quantification of Visual Localization for Autonomous Vehicles](https://mlsanity.com/paper/2506.15851/) (2025)
- [Gaussian Semantic Field for One-shot LiDAR Global Localization](https://mlsanity.com/paper/2510.12101/) (2025)
- [Relational Epipolar Graphs for Robust Relative Camera Pose Estimation](https://mlsanity.com/paper/2604.04554/) (2026)
- [E-S2Feat:Semantic-Guided Spiking Local Feature Detection and Description for Event Cameras](https://mlsanity.com/paper/2608.14027/) (2026)
- [NSFlow: End-to-End Differentiable Neuro-Symbolic Optical Flow for Visual Odometry](https://mlsanity.com/paper/2609.06074/) (2026)

## Abstract (arXiv)

Visual localization is a fundamental task for various applications including autonomous driving and robotics. Prior methods focus on extracting large amounts of often redundant locally reliable features, resulting in limited efficiency and accuracy, especially in large-scale environments under challenging conditions. Instead, we propose to extract globally reliable features by implicitly embedding high-level semantics into both the detection and description processes. Specifically, our semantic-aware detector is able to detect keypoints from reliable regions (e.g. building, traffic lane) and suppress unreliable areas (e.g. sky, car) implicitly instead of relying on explicit semantic labels. This boosts the accuracy of keypoint matching by reducing the number of features sensitive to appearance changes and avoiding the need of additional segmentation networks at test time. Moreover, our descriptors are augmented with semantics and have stronger discriminative ability, providing more inliers at test time. Particularly, experiments on long-term large-scale visual localization Aachen Day-Night and RobotCar-Seasons datasets demonstrate that our model outperforms previous local features and gives competitive accuracy to advanced matchers but is about 2 and 3 times faster when using 2k and 4k keypoints, respectively.