# From Pixels to Objects: A Hierarchical Approach for Part and Object Segmentation Using Local and Global Aggregation

arXiv: https://arxiv.org/abs/2409.01353 · HTML: https://mlsanity.com/paper/2409.01353/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 45 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts](https://mlsanity.com/paper/1406.2031/) (2014)
- [PartImageNet: A Large, High-Quality Dataset of Parts](https://mlsanity.com/paper/2112.00933/) (2021)
- [SegViT: Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2210.05844/) (2022)
- [Superpixel Sampling Networks](https://mlsanity.com/paper/1807.10174/) (2018)
- [Superpixel Segmentation with Fully Convolutional Networks](https://mlsanity.com/paper/2003.12929/) (2020)
- [CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation](https://mlsanity.com/paper/2206.08948/) (2022)
- [Vision Transformer with Super Token Sampling](https://mlsanity.com/paper/2211.11167/) (2022)
- [Image as Set of Points](https://mlsanity.com/paper/2303.01494/) (2023)
- [AtomNAS: Fine-Grained End-to-End Neural Architecture Search](https://mlsanity.com/paper/1912.09640/) (2019)
- [GMNet: Graph Matching Network for Large Scale Part Semantic Segmentation in the Wild](https://mlsanity.com/paper/2007.09073/) (2020)
- [Simple Training Strategies and Model Scaling for Object Detection](https://mlsanity.com/paper/2107.00057/) (2021)
- [Joint Object and Part Segmentation using Deep Learned Potentials](https://mlsanity.com/paper/1505.00276/) (2015)
- [Semantic Segmentation by Early Region Proxy](https://mlsanity.com/paper/2203.14043/) (2022)
- [SPFormer: Enhancing Vision Transformer with Superpixel Representation](https://mlsanity.com/paper/2401.02931/) (2024)
- [kMaX-DeepLab: k-means Mask Transformer](https://mlsanity.com/paper/2207.04044/) (2022)

## Abstract (arXiv)

In this paper, we introduce a hierarchical transformer-based model designed for sophisticated image segmentation tasks, effectively bridging the granularity of part segmentation with the comprehensive scope of object segmentation. At the heart of our approach is a multi-level representation strategy, which systematically advances from individual pixels to superpixels, and ultimately to cohesive group formations. This architecture is underpinned by two pivotal aggregation strategies: local aggregation and global aggregation. Local aggregation is employed to form superpixels, leveraging the inherent redundancy of the image data to produce segments closely aligned with specific parts of the object, guided by object-level supervision. In contrast, global aggregation interlinks these superpixels, organizing them into larger groups that correlate with entire objects and benefit from part-level supervision. This dual aggregation framework ensures a versatile adaptation to varying supervision inputs while maintaining computational efficiency. Our methodology notably improves the balance between adaptability across different supervision modalities and computational manageability, culminating in significant enhancement in segmentation performance. When tested on the PartImageNet dataset, our model achieves a substantial increase, outperforming the previous state-of-the-art by 2.8% and 0.8% in mIoU scores for part and object segmentation, respectively. Similarly, on the Pascal Part dataset, it records performance enhancements of 1.5% and 2.0% for part and object segmentation, respectively.