# Attend and Segment: Attention Guided Active Semantic Segmentation

arXiv: https://arxiv.org/abs/2007.11548 · HTML: https://mlsanity.com/paper/2007.11548/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2020.

## Built on by (most cited first, 9 of 9)

- [Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation](https://mlsanity.com/paper/2207.10436/) (2022)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [Active Visual Exploration Based on Attention-Map Entropy](https://mlsanity.com/paper/2303.06457/) (2023)
- [Consistency driven Sequential Transformers Attention Model for Partially Observable Scenes](https://mlsanity.com/paper/2204.00656/) (2022)
- [Glimpse-Attend-and-Explore: Self-Attention for Active Visual Exploration](https://mlsanity.com/paper/2108.11717/) (2021)
- [AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale](https://mlsanity.com/paper/2404.03482/) (2024)
- [Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos](https://mlsanity.com/paper/2411.08753/) (2024)
- [TORE: Token Recycling in Vision Transformers for Efficient Active Visual Exploration](https://mlsanity.com/paper/2311.15335/) (2023)
- [Switch-a-View: View Selection Learned from Unlabeled In-the-wild Videos](https://mlsanity.com/paper/2412.18386/) (2024)

## Abstract (arXiv)

In a dynamic environment, an agent with a limited field of view/resource cannot fully observe the scene before attempting to parse it. The deployment of common semantic segmentation architectures is not feasible in such settings. In this paper we propose a method to gradually segment a scene given a sequence of partial observations. The main idea is to refine an agent's understanding of the environment by attending the areas it is most uncertain about. Our method includes a self-supervised attention mechanism and a specialized architecture to maintain and exploit spatial memory maps for filling-in the unseen areas in the environment. The agent can select and attend an area while relying on the cues coming from the visited areas to hallucinate the other parts. We reach a mean pixel-wise accuracy of 78.1%, 80.9% and 76.5% on CityScapes, CamVid, and Kitti datasets by processing only 18% of the image pixels (10 retina-like glimpses). We perform an ablation study on the number of glimpses, input image size and effectiveness of retina-like glimpses. We compare our method to several baselines and show that the optimal results are achieved by having access to a very low resolution view of the scene at the first timestep.