mlsanityData to 14 Oct 2026

Understanding Imbalanced Semantic Segmentation Through Neural Collapse

arXiv 2301.01100 ↗First posted 3 Jan 2023cs.CV cs.LGcited by 46

Not on the map. Why?

Abstract (arXiv)

A recent study has shown a phenomenon called neural collapse in that the within-class means of features and the classifier weight vectors converge to the vertices of a simplex equiangular tight frame at the terminal phase of training for classification. In this paper, we explore the corresponding structures of the last-layer feature centers and classifiers in semantic segmentation. Based on our empirical and theoretical analysis, we point out that semantic segmentation naturally brings contextual correlation and imbalanced distribution among classes, which breaks the equiangular and maximally separated structure of neural collapse for both feature centers and classifiers. However, such a symmetric structure is beneficial to discrimination for the minor classes. To preserve these advantages, we introduce a regularizer on feature centers to encourage the network to learn features closer to the appealing structure in imbalanced semantic segmentation. Experimental results show that our method can bring significant improvements on both 2D and 3D semantic segmentation benchmarks. Moreover, our method ranks 1st and sets a new record (+6.8% mIoU) on the ScanNet200 test leaderboard. Code will be available at https://github.com/dvlab-research/Imbalanced-Learning.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 65 of its 86 references that are in the map. First posted on arXiv 3 Jan 2023.

Builds on

65 of 86 references.

References · 65

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  9. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  10. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2020Supervised Contrastive Learningcited by 2,829
38 more
  1. 2016Pyramid Scene Parsing Networkcited by 2,479
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  4. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  5. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
  6. 2020Point Transformercited by 1,470
  7. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  8. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  9. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  10. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  11. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  12. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,133
  13. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  14. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  15. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  16. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  17. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  18. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  19. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  20. 2020Long-tail learning via logit adjustmentcited by 618
  21. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 576
  22. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 554
  23. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 465
  24. 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 400
  25. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  26. 2017Tversky loss function for image segmentation using 3D fully convolutional deep networkscited by 228
  27. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  28. 2017Submanifold Sparse Convolutional Networkscited by 223
  29. 2021Parametric Contrastive Learningcited by 217
  30. 2021Improving Calibration for Long-Tailed Recognitioncited by 212
  31. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  32. 2021Distribution Alignment: A Unified Framework for Long-tail Visual Recognitioncited by 198
  33. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  34. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 167
  35. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  36. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  37. 2022Balanced Contrastive Learning for Long-Tailed Visual Recognitioncited by 144
  38. 2020Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspectivecited by 143

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 19; 2025: 10; 2026: 8.

46 papers.

Sort:

By area:

  1. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  2. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 116
  3. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  4. 2024OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentationcited by 44
  5. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  6. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  7. 2024GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understandingcited by 26
  8. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
  9. 2024DirMixE: Harnessing Test Agnostic Long-tail Recognition with Hierarchical Label Variationscited by 15
  10. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  11. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  12. 2023Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severitycited by 13
34 more
  1. 2024Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapsecited by 13
  2. 2024Contextrast: Contextual Contrastive Learning for Semantic Segmentationcited by 12
  3. 2024FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Datacited by 10
  4. 2024Classes Are Not Equal: An Empirical Study on Image Recognition Fairnesscited by 10
  5. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
  6. 2024Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learningcited by 7
  7. 2024Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretizationcited by 7
  8. 2025Improve Representation for Imbalanced Regression through Geometric Constraintscited by 7
  9. 2024Adaptive Parametric Activation: Unifying and Generalising Activation Functions Across Taskscited by 6
  10. 2025Consistent Supervised-Unsupervised Alignment for Generalized Category Discoverycited by 6
  11. 2024RT-GS2: Real-Time Generalizable Semantic Segmentation for 3D Gaussian Representations of Radiance Fieldscited by 5
  12. 2024Fractal Calibration for long-tailed object detectioncited by 5
  13. 2025BFANet: Revisiting 3D Semantic Segmentation with Boundary Feature Analysiscited by 5
  14. 2024Towards Interpretable Deep Local Learning with Successive Gradient Reconciliationcited by 4
  15. 2024Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspectivecited by 3
  16. 2023Understanding Prompt Tuning for V-L Models Through the Lens of Neural Collapsecited by 2
  17. 2024Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)cited by 2
  18. 2024SGW-based Multi-Task Learning in Vision Taskscited by 2
  19. 2025SRMF: A Data Augmentation and Multimodal Fusion Approach for Long-Tail UHR Satellite Image Segmentationcited by 2
  20. 2023Few-shot 1/a Anomalies Feedback : Damage Vision Mining Opportunity and Embedding Feature Imbalancecited by 1
  21. 2025CLIMB-3D: Continual Learning for Imbalanced 3D Instance Segmentationcited by 1
  22. 2025Segmenting Visuals With Querying Words: Language Anchors For Semi-Supervised Image Segmentationcited by 1
  23. 2025Neural Collapse in Test-Time Adaptationcited by 1
  24. 2026Set Prediction for Next-Day Active Fire Forecastingcited by 1
  25. 2024PAT: Pixel-wise Adaptive Training for Long-tailed Segmentationcited by 0
  26. 2025Enhancing Pre-Trained Model-Based Class-Incremental Learning through Neural Collapsecited by 0
  27. 2025Open-Set Semi-Supervised Learning for Long-Tailed Medical Datasetscited by 0
  28. 2026G2P: Gaussian-to-Point Attribute Alignment for Boundary-Aware 3D Segmentationcited by 0
  29. 2026Coverage-aware Semantic Representation Learning for Underrepresented Visual Conceptscited by 0
  30. 2026HAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learningcited by 0
  31. 2026Entropy Minimization without Model Collapse: Mitigating Prediction Bias in Medical Imagingcited by 0
  32. 2026Class-frequency Guided Noise Schedule for Diffusion Modelscited by 0
  33. 2026Towards Investigating Residual Hearing Loss: Quantification of Fibrosis in a Novel Cochlear OCT Datasetcited by 0
  34. 2026Contextrast++: Robust Multi-Scale Contextual Contrastive Learning for Semantic Segmentationcited by 0