# Isomer: Isomerous Transformer for Zero-shot Video Object Segmentation

arXiv: https://arxiv.org/abs/2308.06693 · HTML: https://mlsanity.com/paper/2308.06693/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 74 references that are in the map.
- First posted on arXiv 13 Aug 2023.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [Structure-measure: A New Way to Evaluate Foreground Maps](https://mlsanity.com/paper/1708.00786/) (2017)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [EfficientFormer: Vision Transformers at MobileNet Speed](https://mlsanity.com/paper/2206.01191/) (2022)
- [Mobile-Former: Bridging MobileNet and Transformer](https://mlsanity.com/paper/2108.05895/) (2021)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks](https://mlsanity.com/paper/2001.06810/) (2020)
- [Suppress and Balance: A Simple Gated Network for Salient Object Detection](https://mlsanity.com/paper/2007.08074/) (2020)
- [Video Object Segmentation with Episodic Graph Memory Networks](https://mlsanity.com/paper/2007.07020/) (2020)

## Built on by (most cited first, 10 of 10)

- [LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation](https://mlsanity.com/paper/2404.19326/) (2024)
- [Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2505.19938/) (2025)
- [Decomposition Betters Tracking Everything Everywhere](https://mlsanity.com/paper/2407.06531/) (2024)
- [GMOS: Grounding Moving Object Segmentation in 3D Space and Time](https://mlsanity.com/paper/2605.30352/) (2026)
- [Treating Motion as Option with Output Selection for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2309.14786/) (2023)
- [GISE-TTT:A Framework for Global InformationSegmentation and Enhancement](https://mlsanity.com/paper/2504.00879/) (2025)
- [Saliency-Motion Guided Trunk-Collateral Network for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2504.05904/) (2025)
- [Temporal Consistency Constrained Transferable Adversarial Attacks with Background Mixup for Action Recognition](https://mlsanity.com/paper/2505.17807/) (2025)
- [FTIO: Frequent Temporally Integrated Objects](https://mlsanity.com/paper/2508.16183/) (2025)
- [DIMOS: Disentangling Instance-level Moving Object Segmentation](https://mlsanity.com/paper/2606.12826/) (2026)

## Abstract (arXiv)

Recent leading zero-shot video object segmentation (ZVOS) works devote to integrating appearance and motion information by elaborately designing feature fusion modules and identically applying them in multiple feature stages. Our preliminary experiments show that with the strong long-range dependency modeling capacity of Transformer, simply concatenating the two modality features and feeding them to vanilla Transformers for feature fusion can distinctly benefit the performance but at a cost of heavy computation. Through further empirical analysis, we find that attention dependencies learned in Transformer in different stages exhibit completely different properties: global query-independent dependency in the low-level stages and semantic-specific dependency in the high-level stages. Motivated by the observations, we propose two Transformer variants: i) Context-Sharing Transformer (CST) that learns the global-shared contextual information within image frames with a lightweight computation. ii) Semantic Gathering-Scattering Transformer (SGST) that models the semantic correlation separately for the foreground and background and reduces the computation cost with a soft token merging mechanism. We apply CST and SGST for low-level and high-level feature fusions, respectively, formulating a level-isomerous Transformer framework for ZVOS task. Compared with the baseline that uses vanilla Transformers for multi-stage fusion, ours significantly increase the speed by 13 times and achieves new state-of-the-art ZVOS performance. Code is available at https://github.com/DLUT-yyc/Isomer.