mlsanityData to 7 Oct 2026

COCO-Stuff: Thing and Stuff Classes in Context

arXiv 1612.03716 ↗First posted 12 Dec 2016cs.CVcited by 858

Not on the map. Why?

Cited by 102 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Semantic classes can be either things (objects with a well-defined shape, e.g. car, person) or stuff (amorphous background regions, e.g. grass, sky). While lots of classification and detection works focus on thing classes, less attention has been given to stuff classes. Nonetheless, stuff classes are important as they allow to explain important aspects of an image, including (1) scene type; (2) which thing classes are likely to be present and their location (through contextual reasoning); (3) physical attributes, material types and geometric properties of the scene. To understand stuff and things in context we introduce COCO-Stuff, which augments all 164K images of the COCO 2017 dataset with pixel-wise annotations for 91 stuff classes. We introduce an efficient stuff annotation protocol based on superpixels, which leverages the original thing annotations. We quantify the speed versus quality trade-off of our protocol and explore the relation between annotation time and boundary complexity. Furthermore, we use COCO-Stuff to analyze: (a) the importance of stuff and thing classes in terms of their surface cover and how frequently they are mentioned in image captions; (b) the spatial relations between stuff and things, highlighting the rich contextual relations that make our dataset unique; (c) the performance of a modern semantic segmentation method on stuff and thing classes, and whether stuff is easier to segment than things.

Key facts. Cited by 858 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 98; 2022: 130; 2023: 190; 2024: 189; 2025: 159; 2026: 91.

858 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  4. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  5. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  6. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  7. 2022Language-driven Semantic Segmentationcited by 711
  8. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  9. 2021Extract Free Dense Labels from CLIPcited by 580
  10. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  11. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  12. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
38 more
  1. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  2. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  3. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  4. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  5. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  6. 2021DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentationcited by 299
  7. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  8. 2022Post-hoc Concept Bottleneck Modelscited by 289
  9. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  10. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  11. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generationcited by 258
  12. 2022SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentationcited by 252
  13. 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
  14. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  15. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  16. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  17. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 219
  18. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 219
  19. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  20. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  21. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  22. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  23. 2021Boundary IoU: Improving Object-Centric Image Segmentation Evaluationcited by 186
  24. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
  25. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  26. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  27. 2021PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clusteringcited by 163
  28. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  29. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  30. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  31. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  32. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  33. 2022Rethinking Semantic Segmentation: A Prototype Viewcited by 140
  34. 2021Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentationcited by 133
  35. 2022Semantic Image Synthesis via Diffusion Modelscited by 131
  36. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  37. 2021Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentationcited by 127
  38. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126

50 most cited of 858 All 858 (JSON)