# Mask2Anomaly: Mask Transformer for Universal Open-set Segmentation

arXiv: https://arxiv.org/abs/2309.04573 · HTML: https://mlsanity.com/paper/2309.04573/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 77 references that are in the map.
- First posted on arXiv 8 Sep 2023.

## Builds on (most cited first, 30 of 56)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)

## Built on by (most cited first, 18 of 18)

- [Open-set Cross Modal Generalization via Multimodal Unified Representation](https://mlsanity.com/paper/2507.14935/) (2025)
- [Open-World Panoptic Segmentation](https://mlsanity.com/paper/2412.12740/) (2024)
- [Benchmarking Vision Foundation Models for Input Monitoring in Autonomous Driving](https://mlsanity.com/paper/2501.08083/) (2025)
- [Panoramic Out-of-Distribution Segmentation](https://mlsanity.com/paper/2505.03539/) (2025)
- [Out-of-Distribution Semantic Occupancy Prediction](https://mlsanity.com/paper/2506.21185/) (2025)
- [OoDDINO:A Multi-level Framework for Anomaly Segmentation on Complex Road Scenes](https://mlsanity.com/paper/2507.01455/) (2025)
- [Stereo-based 3D Anomaly Object Detection for Autonomous Driving: A New Dataset and Baseline](https://mlsanity.com/paper/2507.09214/) (2025)
- [Road Obstacle Video Segmentation](https://mlsanity.com/paper/2509.13181/) (2025)
- [Open-set Anomaly Segmentation in Complex Scenarios](https://mlsanity.com/paper/2504.19706/) (2025)
- [A Gift from the Integration of Discriminative and Diffusion-based Generative Learning: Boundary Refinement Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2507.01573/) (2025)
- [Leveraging Text-Driven Semantic Variation for Robust OOD Segmentation](https://mlsanity.com/paper/2511.07238/) (2025)
- [Training-Free Out-Of-Distribution Segmentation With Foundation Models](https://mlsanity.com/paper/2510.02909/) (2025)
- [Language-Guided Open-World Anomaly Segmentation](https://mlsanity.com/paper/2512.01427/) (2025)
- [ClimaOoD: Improving Anomaly Segmentation via Physically Realistic Synthetic Data](https://mlsanity.com/paper/2512.02686/) (2025)
- [ProOOD: Prototype-Guided Out-of-Distribution 3D Occupancy Prediction](https://mlsanity.com/paper/2604.01081/) (2026)
- [Hyp2Former: Hierarchy-Aware Hyperbolic Embeddings for Open-Set Panoptic Segmentation](https://mlsanity.com/paper/2605.02580/) (2026)
- [Road-Aware Anomaly Segmentation with Query-Guided Polygons and CLIP in Autonomous Driving](https://mlsanity.com/paper/2607.04304/) (2026)
- [LARAD: Layout-Aware Road Anomaly Detection via Spatial-Logic Reasoning](https://mlsanity.com/paper/2607.12858/) (2026)

## Abstract (arXiv)

Segmenting unknown or anomalous object instances is a critical task in autonomous driving applications, and it is approached traditionally as a per-pixel classification problem. However, reasoning individually about each pixel without considering their contextual semantics results in high uncertainty around the objects' boundaries and numerous false positives. We propose a paradigm change by shifting from a per-pixel classification to a mask classification. Our mask-based method, Mask2Anomaly, demonstrates the feasibility of integrating a mask-classification architecture to jointly address anomaly segmentation, open-set semantic segmentation, and open-set panoptic segmentation. Mask2Anomaly includes several technical novelties that are designed to improve the detection of anomalies/unknown objects: i) a global masked attention module to focus individually on the foreground and background regions; ii) a mask contrastive learning that maximizes the margin between an anomaly and known classes; iii) a mask refinement solution to reduce false positives; and iv) a novel approach to mine unknown instances based on the mask-architecture properties. By comprehensive qualitative and qualitative evaluation, we show Mask2Anomaly achieves new state-of-the-art results across the benchmarks of anomaly segmentation, open-set semantic segmentation, and open-set panoptic segmentation.