# Hierarchical Feature Alignment Network for Unsupervised Video Object Segmentation

arXiv: https://arxiv.org/abs/2207.08485 · HTML: https://mlsanity.com/paper/2207.08485/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 80 references that are in the map.
- First posted on arXiv 18 Jul 2022.

## Builds on (most cited first, 30 of 56)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Align Deep Features for Oriented Object Detection](https://mlsanity.com/paper/2008.09397/) (2020)
- [Video Object Segmentation with Language Referring Expressions](https://mlsanity.com/paper/1803.08006/) (2018)
- [See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks](https://mlsanity.com/paper/2001.06810/) (2020)
- [Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion](https://mlsanity.com/paper/2103.07941/) (2021)
- [Semantic Flow for Fast and Accurate Scene Parsing](https://mlsanity.com/paper/2002.10120/) (2020)
- [Video Object Segmentation with Episodic Graph Memory Networks](https://mlsanity.com/paper/2007.07020/) (2020)
- [SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation](https://mlsanity.com/paper/2101.08833/) (2021)
- [RGB-Infrared Cross-Modality Person Re-Identification via Joint Pixel and Feature Alignment](https://mlsanity.com/paper/1910.05839/) (2019)
- [Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement](https://mlsanity.com/paper/2010.07958/) (2020)
- [Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2103.14581/) (2021)
- [SegFlow: Joint Learning for Video Object Segmentation and Optical Flow](https://mlsanity.com/paper/1709.06750/) (2017)
- [RVOS: End-to-End Recurrent Network for Video Object Segmentation](https://mlsanity.com/paper/1903.05612/) (2019)
- [Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks](https://mlsanity.com/paper/2001.06807/) (2020)
- [FaPN: Feature-aligned Pyramid Network for Dense Image Prediction](https://mlsanity.com/paper/2108.07058/) (2021)
- [Jo-SRC: A Contrastive Approach for Combating Noisy Labels](https://mlsanity.com/paper/2103.13029/) (2021)

## Built on by (most cited first, 30 of 37)

- [LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation](https://mlsanity.com/paper/2404.19326/) (2024)
- [FECANet: Boosting Few-Shot Semantic Segmentation with Feature-Enhanced Context-Aware Network](https://mlsanity.com/paper/2301.08160/) (2023)
- [VideoMAC: Video Masked Autoencoders Meet ConvNets](https://mlsanity.com/paper/2402.19082/) (2024)
- [SegDINO: An Efficient Design for Medical and Natural Image Segmentation with DINO-V3](https://mlsanity.com/paper/2509.00833/) (2025)
- [Guided Slot Attention for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2303.08314/) (2023)
- [SALI: Short-term Alignment and Long-term Interaction Network for Colonoscopy Video Polyp Segmentation](https://mlsanity.com/paper/2406.13532/) (2024)
- [Spatial Structure Constraints for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2401.11122/) (2024)
- [Dual Prototype Attention for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2211.12036/) (2022)
- [UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model](https://mlsanity.com/paper/2305.12659/) (2023)
- [Isomer: Isomerous Transformer for Zero-shot Video Object Segmentation](https://mlsanity.com/paper/2308.06693/) (2023)
- [Co-attention Propagation Network for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2304.03910/) (2023)
- [Event-Free Moving Object Segmentation from Moving Ego Vehicle](https://mlsanity.com/paper/2305.00126/) (2023)
- [Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning](https://mlsanity.com/paper/2503.17914/) (2025)
- [SMITE: Segment Me In TimE](https://mlsanity.com/paper/2410.18538/) (2024)
- [Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection](https://mlsanity.com/paper/2503.17080/) (2025)
- [Semi-Supervised Semantic Segmentation With Region Relevance](https://mlsanity.com/paper/2304.11539/) (2023)
- [Learning Motion and Temporal Cues for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2501.07806/) (2025)
- [Efficient Token Compression for Vision Transformer with Spatial Information Preserved](https://mlsanity.com/paper/2503.23455/) (2025)
- [Relating CNN-Transformer Fusion Network for Change Detection](https://mlsanity.com/paper/2407.03178/) (2024)
- [Hierarchical Graph Pattern Understanding for Zero-Shot VOS](https://mlsanity.com/paper/2312.09525/) (2023)
- [Domain Adaptation for Large-Vocabulary Object Detectors](https://mlsanity.com/paper/2401.06969/) (2024)
- [Holistic Prototype Attention Network for Few-Shot VOS](https://mlsanity.com/paper/2307.07933/) (2023)
- [Sketch-based Video Object Segmentation: Benchmark and Analysis](https://mlsanity.com/paper/2311.07261/) (2023)
- [TransFlow: Motion Knowledge Transfer from Video Diffusion Models to Video Salient Object Detection](https://mlsanity.com/paper/2507.19789/) (2025)
- [Towards Remote Sensing Change Detection with Neural Memory](https://mlsanity.com/paper/2602.10491/) (2026)
- [GMOS: Grounding Moving Object Segmentation in 3D Space and Time](https://mlsanity.com/paper/2605.30352/) (2026)
- [Attention Map Guided Transformer Pruning for Edge Device](https://mlsanity.com/paper/2304.01452/) (2023)
- [Towards Imbalanced Motion: Part-Decoupling Network for Video Portrait Segmentation](https://mlsanity.com/paper/2307.16565/) (2023)
- [Treating Motion as Option with Output Selection for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2309.14786/) (2023)
- [Unsupervised Region-Growing Network for Object Segmentation in Atmospheric Turbulence](https://mlsanity.com/paper/2311.03572/) (2023)

## Abstract (arXiv)

Optical flow is an easily conceived and precious cue for advancing unsupervised video object segmentation (UVOS). Most of the previous methods directly extract and fuse the motion and appearance features for segmenting target objects in the UVOS setting. However, optical flow is intrinsically an instantaneous velocity of all pixels among consecutive frames, thus making the motion features not aligned well with the primary objects among the corresponding frames. To solve the above challenge, we propose a concise, practical, and efficient architecture for appearance and motion feature alignment, dubbed hierarchical feature alignment network (HFAN). Specifically, the key merits in HFAN are the sequential Feature AlignMent (FAM) module and the Feature AdaptaTion (FAT) module, which are leveraged for processing the appearance and motion features hierarchically. FAM is capable of aligning both appearance and motion features with the primary object semantic representations, respectively. Further, FAT is explicitly designed for the adaptive fusion of appearance and motion features to achieve a desirable trade-off between cross-modal features. Extensive experiments demonstrate the effectiveness of the proposed HFAN, which reaches a new state-of-the-art performance on DAVIS-16, achieving 88.7 J&F Mean, i.e., a relative improvement of 3.5% over the best published result.