# Image as Set of Points

arXiv: https://arxiv.org/abs/2303.01494 · HTML: https://mlsanity.com/paper/2303.01494/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 67 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)

## Built on by (most cited first, 30 of 43)

- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding](https://mlsanity.com/paper/2410.17434/) (2024)
- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://mlsanity.com/paper/2406.12708/) (2024)
- [An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels](https://mlsanity.com/paper/2406.09415/) (2024)
- [DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving](https://mlsanity.com/paper/2412.09043/) (2024)
- [DVLO: Deep Visual-LiDAR Odometry with Local-to-Global Feature Fusion and Bi-Directional Structure Alignment](https://mlsanity.com/paper/2403.18274/) (2024)
- [Context-Aware Interaction Network for RGB-T Semantic Segmentation](https://mlsanity.com/paper/2401.01624/) (2024)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [SPFormer: Enhancing Vision Transformer with Superpixel Representation](https://mlsanity.com/paper/2401.02931/) (2024)
- [Neural Clustering based Visual Representation Learning](https://mlsanity.com/paper/2403.17409/) (2024)
- [ASY-VRNet: Waterway Panoptic Driving Perception Model based on Asymmetric Fair Fusion of Vision and 4D mmWave Radar](https://mlsanity.com/paper/2308.10287/) (2023)
- [Dual-stage Hyperspectral Image Classification Model with Spectral Supertoken](https://mlsanity.com/paper/2407.07307/) (2024)
- [3D Occupancy Prediction with Low-Resolution Queries via Prototype-aware View Transformation](https://mlsanity.com/paper/2503.15185/) (2025)
- [Post-hoc Part-prototype Networks](https://mlsanity.com/paper/2406.03421/) (2024)
- [Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokens](https://mlsanity.com/paper/2412.04680/) (2024)
- [Revisiting the Encoding of Satellite Image Time Series](https://mlsanity.com/paper/2305.02086/) (2023)
- [ClusterFormer: Clustering As A Universal Visual Learner](https://mlsanity.com/paper/2309.13196/) (2023)
- [DVHGNN: Multi-Scale Dilated Vision HGNN for Efficient Vision Recognition](https://mlsanity.com/paper/2503.14867/) (2025)
- [One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding](https://mlsanity.com/paper/2604.14149/) (2026)
- [Improving Scene Graph Generation with Superpixel-Based Interaction Learning](https://mlsanity.com/paper/2308.02339/) (2023)
- [Perceptual Group Tokenizer: Building Perception with Iterative Grouping](https://mlsanity.com/paper/2311.18296/) (2023)
- [Prototypical Transformer as Unified Motion Learners](https://mlsanity.com/paper/2406.01559/) (2024)
- [Merging Context Clustering with Visual State Space Models for Medical Image Segmentation](https://mlsanity.com/paper/2501.01618/) (2025)
- [PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Models](https://mlsanity.com/paper/2510.22936/) (2025)
- [Sparse Refinement for Efficient High-Resolution Semantic Segmentation](https://mlsanity.com/paper/2407.19014/) (2024)
- [PoCo: Point Context Cluster for RGBD Indoor Place Recognition](https://mlsanity.com/paper/2404.02885/) (2024)
- [Flexible ViG: Learning the Self-Saliency for Flexible Object Recognition](https://mlsanity.com/paper/2406.18585/) (2024)
- [URLOST: Unsupervised Representation Learning without Stationarity or Topology](https://mlsanity.com/paper/2310.04496/) (2023)
- [CSCPR: Cross-Source-Context Indoor RGB-D Place Recognition](https://mlsanity.com/paper/2407.17457/) (2024)
- [Mammo-Clustering: A Multi-views Tri-level Information Fusion Context Clustering Framework for Localization and Classification in Mammography](https://mlsanity.com/paper/2409.14876/) (2024)

## Abstract (arXiv)

What is an image and how to extract latent features? Convolutional Networks (ConvNets) consider an image as organized pixels in a rectangular shape and extract features via convolutional operation in local region; Vision Transformers (ViTs) treat an image as a sequence of patches and extract features via attention mechanism in a global range. In this work, we introduce a straightforward and promising paradigm for visual representation, which is called Context Clusters. Context clusters (CoCs) view an image as a set of unorganized points and extract features via simplified clustering algorithm. In detail, each point includes the raw feature (e.g., color) and positional information (e.g., coordinates), and a simplified clustering algorithm is employed to group and extract deep features hierarchically. Our CoCs are convolution- and attention-free, and only rely on clustering algorithm for spatial interaction. Owing to the simple design, we show CoCs endow gratifying interpretability via the visualization of clustering process. Our CoCs aim at providing a new perspective on image and visual representation, which may enjoy broad applications in different domains and exhibit profound insights. Even though we are not targeting SOTA performance, COCs still achieve comparable or even better results than ConvNets or ViTs on several benchmarks. Codes are available at: https://github.com/ma-xu/Context-Cluster.