# Segment Anything in High Quality

arXiv: https://arxiv.org/abs/2306.01567 · HTML: https://mlsanity.com/paper/2306.01567/

## Key facts

- Cited by 376 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 59 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Generalized Decoding for Pixel, Image, and Language](https://mlsanity.com/paper/2212.11270/) (2022)
- [ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://mlsanity.com/paper/1704.08545/) (2017)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [Boundary IoU: Improving Object-Centric Image Segmentation Evaluation](https://mlsanity.com/paper/2103.16562/) (2021)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)

## Built on by (most cited first, 30 of 376)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model](https://mlsanity.com/paper/2306.16269/) (2023)
- [Osprey: Pixel Understanding with Visual Instruction Tuning](https://mlsanity.com/paper/2312.10032/) (2023)
- [RepViT: Revisiting Mobile CNN From ViT Perspective](https://mlsanity.com/paper/2307.09283/) (2023)
- [Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization](https://mlsanity.com/paper/2311.06243/) (2023)
- [SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding](https://mlsanity.com/paper/2310.15308/) (2023)
- [MedSAM2: Segment Anything in 3D Medical Images and Videos](https://mlsanity.com/paper/2504.03600/) (2025)
- [SAI3D: Segment Any Instance in 3D Scenes](https://mlsanity.com/paper/2312.11557/) (2023)
- [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023)
- [VRP-SAM: SAM with Visual Reference Prompt](https://mlsanity.com/paper/2402.17726/) (2024)
- [VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM](https://mlsanity.com/paper/2501.00599/) (2024)
- [CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor](https://mlsanity.com/paper/2312.07661/) (2023)
- [EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model](https://mlsanity.com/paper/2406.20076/) (2024)
- [DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos](https://mlsanity.com/paper/2405.02280/) (2024)
- [ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions](https://mlsanity.com/paper/2401.10232/) (2024)
- [Watermark Anything with Localized Messages](https://mlsanity.com/paper/2411.07231/) (2024)
- [LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning](https://mlsanity.com/paper/2404.08767/) (2024)
- [Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos](https://mlsanity.com/paper/2506.05302/) (2025)
- [Matching Anything by Segmenting Anything](https://mlsanity.com/paper/2406.04221/) (2024)
- [RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation](https://mlsanity.com/paper/2402.15487/) (2024)
- [DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion](https://mlsanity.com/paper/2407.12899/) (2024)
- [Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding](https://mlsanity.com/paper/2312.00081/) (2023)
- [IRSAM: Advancing Segment Anything Model for Infrared Small Target Detection](https://mlsanity.com/paper/2407.07520/) (2024)
- [PosterMaker: Towards High-Quality Product Poster Generation with Accurate Text Rendering](https://mlsanity.com/paper/2504.06632/) (2025)

## Abstract (arXiv)

The recent Segment Anything Model (SAM) represents a big leap in scaling up segmentation models, allowing for powerful zero-shot capabilities and flexible prompting. Despite being trained with 1.1 billion masks, SAM's mask prediction quality falls short in many cases, particularly when dealing with objects that have intricate structures. We propose HQ-SAM, equipping SAM with the ability to accurately segment any object, while maintaining SAM's original promptable design, efficiency, and zero-shot generalizability. Our careful design reuses and preserves the pre-trained model weights of SAM, while only introducing minimal additional parameters and computation. We design a learnable High-Quality Output Token, which is injected into SAM's mask decoder and is responsible for predicting the high-quality mask. Instead of only applying it on mask-decoder features, we first fuse them with early and final ViT features for improved mask details. To train our introduced learnable parameters, we compose a dataset of 44K fine-grained masks from several sources. HQ-SAM is only trained on the introduced detaset of 44k masks, which takes only 4 hours on 8 GPUs. We show the efficacy of HQ-SAM in a suite of 10 diverse segmentation datasets across different downstream tasks, where 8 out of them are evaluated in a zero-shot transfer protocol. Our code and pretrained models are at https://github.com/SysCV/SAM-HQ.