mlsanityData to 7 Oct 2026

Removing supervision in semantic segmentation with local-global matching and area balancing

arXiv 2303.17410 ↗First posted 30 Mar 2023cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Removing supervision in semantic segmentation is still tricky. Current approaches can deal with common categorical patterns yet resort to multi-stage architectures. We design a novel end-to-end model leveraging local-global patch matching to predict categories, good localization, area and shape of objects for semantic segmentation. The local-global matching is, in turn, compelled by optimal transport plans fulfilling area constraints nearing a solution for exact shape prediction. Our model attains state-of-the-art in Weakly Supervised Semantic Segmentation, only image-level labels, with 75% mIoU on PascalVOC2012 val set and 46% on MS-COCO2014 val set. Dropping the image-level labels and clustering self-supervised learned features to yield pseudo-multi-level labels, we obtain an unsupervised model for semantic segmentation. We also attain state-of-the-art on Unsupervised Semantic Segmentation with 43.6% mIoU on PascalVOC2012 val set and 19.4% on MS-COCO2014 val set. The code is available at https://github.com/deepplants/PC2M.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 54 of its 69 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

54 of 69 references.

References · 54

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  8. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  10. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  11. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  12. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
38 more
  1. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  2. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
  3. 2018Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentationcited by 321
  4. 2020Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentationcited by 299
  5. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  6. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  7. 2017Near-linear time approximation algorithms for optimal transport via Sinkhorn iterationcited by 219
  8. 2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
  9. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  10. 2021PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clusteringcited by 163
  11. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  12. 2020Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentationcited by 144
  13. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  14. 2021Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentationcited by 134
  15. 2022Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformerscited by 118
  16. 2018Computational Optimal Transport: Complexity by Accelerated Gradient Descent Is Better Than by Sinkhorn's Algorithmcited by 111
  17. 2022Self-Supervised Learning of Object Parts for Semantic Segmentationcited by 90
  18. 2022Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentationcited by 87
  19. 2021Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrastcited by 85
  20. 2021Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentationcited by 77
  21. 2021Complementary Patch for Weakly Supervised Semantic Segmentationcited by 75
  22. 2022Class Re-Activation Maps for Weakly-Supervised Semantic Segmentationcited by 75
  23. 2022Cross Language Image Matching for Weakly Supervised Semantic Segmentationcited by 72
  24. 2022Self-Supervised Visual Representation Learning with Semantic Groupingcited by 70
  25. 2021Reducing Information Bottleneck for Weakly Supervised Semantic Segmentationcited by 65
  26. 2022Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentationcited by 64
  27. 2020Unsupervised Learning of Image Segmentation Based on Differentiable Feature Clusteringcited by 61
  28. 2020Autoregressive Unsupervised Image Segmentationcited by 60
  29. 2022Weakly Supervised Semantic Segmentation using Out-of-Distribution Datacited by 58
  30. 2021Revisiting Contrastive Methods for Unsupervised Learning of Visual Representationscited by 48
  31. 2022Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentationcited by 41
  32. 2022Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformerscited by 39
  33. 2019DeepUSPS: Deep Robust Unsupervised Saliency Prediction With Self-Supervisioncited by 38
  34. 2021TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentationcited by 38
  35. 2021Exploring Set Similarity for Dense Self-supervised Representation Learningcited by 29
  36. 2021Activation Modulation and Recalibration Scheme for Weakly Supervised Semantic Segmentationcited by 29
  37. 2018Semi-dual Regularized Optimal Transportcited by 26
  38. 2021Dense Semantic Contrast for Self-Supervised Visual Representation Learningcited by 24

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.