# LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation

arXiv: https://arxiv.org/abs/2606.11837 · HTML: https://mlsanity.com/paper/2606.11837/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 51 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2302.12242/) (2023)
- [Generalized Decoding for Pixel, Image, and Language](https://mlsanity.com/paper/2212.11270/) (2022)
- [CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2303.11797/) (2023)
- [A Simple Framework for Open-Vocabulary Segmentation and Detection](https://mlsanity.com/paper/2303.08131/) (2023)
- [Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut](https://mlsanity.com/paper/2202.11539/) (2022)
- [Open-Vocabulary Universal Image Segmentation with MaskCLIP](https://mlsanity.com/paper/2208.08984/) (2022)
- [Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning](https://mlsanity.com/paper/2212.04994/) (2022)
- [ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2408.04883/) (2024)
- [SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2311.15537/) (2023)
- [FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation](https://mlsanity.com/paper/2303.17225/) (2023)
- [Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval](https://mlsanity.com/paper/1904.03451/) (2019)
- [Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively](https://mlsanity.com/paper/2401.02955/) (2024)
- [Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval](https://mlsanity.com/paper/1703.05605/) (2017)
- [SketchyCOCO: Image Generation from Freehand Scene Sketches](https://mlsanity.com/paper/2003.02683/) (2020)
- [FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context](https://mlsanity.com/paper/2203.02113/) (2022)
- [Open-Vocabulary Segmentation with Semantic-Assisted Calibration](https://mlsanity.com/paper/2312.04089/) (2023)
- [Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation](https://mlsanity.com/paper/2404.06542/) (2024)
- [Open-vocabulary Panoptic Segmentation with Embedding Modulation](https://mlsanity.com/paper/2303.11324/) (2023)
- [Open-Vocabulary Semantic Segmentation with Image Embedding Balancing](https://mlsanity.com/paper/2406.09829/) (2024)
- [CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2411.10086/) (2024)
- [SketchyScene: Richly-Annotated Scene Sketches](https://mlsanity.com/paper/1808.02473/) (2018)
- [LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentation](https://mlsanity.com/paper/2503.19777/) (2025)

## Abstract (arXiv)

Open-vocabulary scene sketch semantic segmentation aims to assign dense semantic labels to sparse line drawings based on flexible category vocabularies specified at inference time, without relying on pixel-level annotations during training. Unlike natural images, sketches lack texture and color cues, making semantic understanding heavily dependent on stroke layout and spatial configuration, a challenge that renders single-layer vision-language features inherently unstable. Our key observation is that attention maps from different Vision Transformer layers encode complementary spatial cues: shallow layers capture global structural layouts, while deeper layers focus on local stroke intersections and object parts. This suggests that cross-layer aggregation provides a more robust structural prior than any individual layer alone. Leveraging this insight, we propose a structure-aware framework built upon Layer-wise Accumulated Structural Attention (LASA), which aggregates multi-layer attention to guide hierarchical semantic alignment under weak supervision and refine predictions during inference. Experiments on FS-COCO, SFSD, and FrISS show that LASA improves mIoU by +3.43, +8.01, and +15.74 over the prior weakly supervised baselines, demonstrating consistent gains in both segmentation accuracy and spatial coherence. Our source code will be made publicly available.