# AIMS: All-Inclusive Multi-Level Segmentation

arXiv: https://arxiv.org/abs/2305.17768 · HTML: https://mlsanity.com/paper/2305.17768/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 59 references that are in the map.
- First posted on arXiv 28 May 2023.

## Builds on (most cited first, 30 of 44)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning](https://mlsanity.com/paper/2304.03411/) (2023)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2302.12242/) (2023)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space](https://mlsanity.com/paper/2103.06030/) (2021)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [PACO: Parts and Attributes of Common Objects](https://mlsanity.com/paper/2301.01795/) (2023)
- [Simple multi-dataset detection](https://mlsanity.com/paper/2102.13086/) (2021)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes](https://mlsanity.com/paper/1707.09465/) (2017)
- [Fully Convolutional Networks for Panoptic Segmentation](https://mlsanity.com/paper/2012.00720/) (2020)
- [A Survey on Instance Segmentation: State of the art](https://mlsanity.com/paper/2007.00047/) (2020)
- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)

## Built on by (most cited first, 6 of 6)

- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Osprey: Pixel Understanding with Visual Instruction Tuning](https://mlsanity.com/paper/2312.10032/) (2023)
- [Multi-Modal Prototypes for Open-World Semantic Segmentation](https://mlsanity.com/paper/2307.02003/) (2023)
- [High-Quality Entity Segmentation and Grounding](https://mlsanity.com/paper/2402.02555/) (2024)
- [TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation](https://mlsanity.com/paper/2312.06630/) (2023)
- [Effective Adapter for Face Recognition in the Wild](https://mlsanity.com/paper/2312.01734/) (2023)

## Abstract (arXiv)

Despite the progress of image segmentation for accurate visual entity segmentation, completing the diverse requirements of image editing applications for different-level region-of-interest selections remains unsolved. In this paper, we propose a new task, All-Inclusive Multi-Level Segmentation (AIMS), which segments visual regions into three levels: part, entity, and relation (two entities with some semantic relationships). We also build a unified AIMS model through multi-dataset multi-task training to address the two major challenges of annotation inconsistency and task correlation. Specifically, we propose task complementarity, association, and prompt mask encoder for three-level predictions. Extensive experiments demonstrate the effectiveness and generalization capacity of our method compared to other state-of-the-art methods on a single dataset or the concurrent work on segmenting anything. We will make our code and training model publicly available.