mlsanityData to 7 Oct 2026

ClusterFormer: Clustering As A Universal Visual Learner

arXiv 2309.13196 ↗First posted 22 Sep 2023cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper presents CLUSTERFORMER, a universal vision model that is based on the CLUSTERing paradigm with TransFORMER. It comprises two novel designs: 1. recurrent cross-attention clustering, which reformulates the cross-attention mechanism in Transformer and enables recursive updates of cluster centers to facilitate strong representation learning; and 2. feature dispatching, which uses the updated cluster centers to redistribute image features through similarity-based metrics, resulting in a transparent pipeline. This elegant design streamlines an explainable and transferable workflow, capable of tackling heterogeneous vision tasks (i.e., image classification, object detection, and image segmentation) with varying levels of clustering granularity (i.e., image-, box-, and pixel-level). Empirical results demonstrate that CLUSTERFORMER outperforms various well-known specialized architectures, achieving 83.41% top-1 acc. over ImageNet-1K for image classification, 54.2% and 47.0% mAP over MS COCO for object detection and instance segmentation, 52.4% mIoU over ADE20K for semantic segmentation, and 55.8% PQ over COCO Panoptic for panoptic segmentation. For its efficacy, we hope our work can catalyze a paradigm shift in universal models in computer vision.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 65 of its 103 references that are in the map. First posted on arXiv 22 Sep 2023.

Builds on

65 of 103 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2017Mask R-CNNcited by 7,259
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
38 more
  1. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  2. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  3. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  4. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  8. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  9. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  10. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  11. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  12. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  13. 2018Panoptic Segmentationcited by 796
  14. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  15. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  16. 2019Panoptic Feature Pyramid Networkscited by 644
  17. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  18. 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 494
  19. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  20. 2019PointRend: Image Segmentation as Renderingcited by 439
  21. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
  22. 2021Conditional DETR for Fast Training Convergencecited by 422
  23. 2019Panoptic-DeepLabcited by 362
  24. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  25. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  26. 2021K-Net: Towards Unified Image Segmentationcited by 211
  27. 2022Vision GNN: An Image is Worth Graph of Nodescited by 185
  28. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  29. 2021Instances as Queriescited by 157
  30. 2020BlendMask: Top-Down Meets Bottom-Up for Instance Segmentationcited by 144
  31. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  32. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  33. 2021Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsitycited by 98
  34. 2020Fully Convolutional Networks for Panoptic Segmentationcited by 92
  35. 2021SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentationcited by 87
  36. 2018Grid R-CNNcited by 85
  37. 2018Superpixel Sampling Networkscited by 79
  38. 2021SOLQ: Segmenting Objects by Learning Queriescited by 68

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025DAMamba: Vision State Space Model with Dynamic Adaptive Scancited by 9
  2. 2025DVHGNN: Multi-Scale Dilated Vision HGNN for Efficient Vision Recognitioncited by 6
  3. 2025CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 6
  4. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networkscited by 3
  5. 2025Native Segmentation Vision Transformerscited by 2
  6. 2026Deep Sprite-based Image Models: An Analysiscited by 0