mlsanityData to 7 Oct 2026

Language-driven Semantic Segmentation

arXiv 2201.03546 ↗First posted 10 Jan 2022cs.CV cs.CL cs.LGcited by 711

Not on the map. Why?

Cited by 103 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present LSeg, a novel model for language-driven semantic image segmentation. LSeg uses a text encoder to compute embeddings of descriptive input labels (e.g., "grass" or "building") together with a transformer-based image encoder that computes dense per-pixel embeddings of the input image. The image encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class. The text embeddings provide a flexible label representation in which semantically similar labels map to similar regions in the embedding space (e.g., "cat" and "furry"). This allows LSeg to generalize to previously unseen categories at test time, without retraining or even requiring a single additional training sample. We demonstrate that our approach achieves highly competitive zero-shot performance compared to existing zero- and few-shot semantic segmentation methods, and even matches the accuracy of traditional segmentation algorithms when a fixed label set is provided. Code and demo are available at https://github.com/isl-org/lang-seg.

Key facts. Cited by 711 papers in the map (data through 7 Oct 2026). Builds on 38 of its 55 references that are in the map. First posted on arXiv 10 Jan 2022.

Builds on

38 of 55 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  9. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  10. 2021Vision Transformers for Dense Predictioncited by 1,789
  11. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
26 more
  1. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  2. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  3. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  4. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  5. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  6. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  7. 2019PANet: Few-Shot Image Semantic Segmentation with Prototype Alignmentcited by 449
  8. 2020ResNeSt: Split-Attention Networkscited by 445
  9. 2017One-Shot Learning for Semantic Segmentationcited by 337
  10. 2018Recent Advances in Open Set Recognition: A Surveycited by 309
  11. 2020Prior Guided Feature Enrichment Network for Few-Shot Segmentationcited by 271
  12. 2019Zero-Shot Semantic Segmentationcited by 245
  13. 2021Hypercorrelation Squeeze for Few-Shot Segmentationcited by 193
  14. 2019Feature Weighting and Boosting for Few-Shot Segmentationcited by 189
  15. 2018IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environmentscited by 163
  16. 2020Part-aware Prototype Network for Few-shot Semantic Segmentationcited by 162
  17. 2019FSS-1000: A 1000-Class Dataset for Few-Shot Segmentationcited by 160
  18. 2020Few-Shot Segmentation Without Meta-Learning: A Good Transductive Inference Is All You Need?cited by 135
  19. 2021MSeg: A Composite Dataset for Multi-domain Semantic Segmentationcited by 122
  20. 2021Learning Placeholders for Open-Set Recognitioncited by 121
  21. 2020Hybrid Models for Open Set Recognitioncited by 109
  22. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  23. 2019Adaptive Context Network for Scene Parsingcited by 51
  24. 2020Few-Shot Open-Set Recognition using Meta-Learningcited by 50
  25. 2019Positional Normalizationcited by 28
  26. 2021Exploring Robustness of Unsupervised Domain Adaptation in Semantic Segmentationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 44; 2023: 189; 2024: 192; 2025: 167; 2026: 119.

711 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  5. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  6. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  7. 2023Segment Everything Everywhere All at Oncecited by 546
  8. 2023LERF: Language Embedded Radiance Fieldscited by 538
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  11. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  12. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
38 more
  1. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  2. 2022Visual Language Maps for Robot Navigationcited by 362
  3. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  4. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  5. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  6. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  7. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  8. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  9. 2023Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspectivecited by 276
  10. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  11. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  12. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  13. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  14. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  15. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  16. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  17. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  18. 2023Teaching CLIP to Count to Tencited by 194
  19. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  20. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  21. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  22. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  23. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 171
  24. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  25. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  26. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 141
  27. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  28. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  29. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  30. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  31. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  32. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  33. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  34. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  35. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  36. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 115
  37. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  38. 2024GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editingcited by 114

50 most cited of 711 All 711 (JSON)