# Modeling Long-Term Memory and Temporal Attention Shifts for Video Salient Object Ranking with a New Benchmark

arXiv: https://arxiv.org/abs/2203.17257 · HTML: https://mlsanity.com/paper/2203.17257/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 28 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 9 of 9)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition](https://mlsanity.com/paper/1312.7570/) (2013)
- [Revisiting Salient Object Detection: Simultaneous Detection, Ranking, and Subitizing of Multiple Salient Objects](https://mlsanity.com/paper/1803.05082/) (2018)
- [Salient Object Ranking with Position-Preserved Attention](https://mlsanity.com/paper/2106.05047/) (2021)
- [Instance-Level Relative Saliency Ranking with Graph Reasoning](https://mlsanity.com/paper/2107.03824/) (2021)
- [Bi-directional Object-context Prioritization Learning for Saliency Ranking](https://mlsanity.com/paper/2203.09416/) (2022)

## Built on by (most cited first, 2 of 2)

- [Rethinking Object Saliency Ranking: A Novel Whole-flow Processing Paradigm](https://mlsanity.com/paper/2312.03226/) (2023)
- [Leverage Task Context for Object Affordance Ranking](https://mlsanity.com/paper/2411.16082/) (2024)

## Abstract (arXiv)

Salient Object Ranking (SOR) aims to estimate the relative saliency order among multiple salient objects. While SOR has been extensively studied in static images, Video Salient Object Ranking (VSOR) remains largely underexplored due to the lack of effective temporal saliency modeling. In particular, existing VSOR methods rely on short input frame clips, which limits their ability to capture long-term saliency evolution and identify dynamic attention shifts. To address these challenges, we propose LoTAS, a long-term memory framework for VSOR that jointly models historical saliency states and temporal attention transitions. To model historical saliency, we propose a Temporal Context Decoder (TCD) and a Rank-aware Saliency State Encoder (RSSE). The TCD retrieves historical saliency states from memory queries to provide references to previously salient instances and long-range temporal context, while the RSSE encodes current predictions into rank-aware state embeddings and updates the memory for future frames, allowing reliable ranking cues to accumulate across long video sequences. To capture temporal attention transitions, we introduce explicit inter-frame rank-transition supervision and jointly learn a binary transition predictor as an auxiliary task alongside ordinal ranking. In addition, to address the limited video types and scene diversity in the existing VSOR dataset, we propose a challenging dataset that covers diverse video types and scenes with 124 videos and 16,610 frames. Experimental results demonstrate that our method outperforms state-of-the-art VSOR methods. We will make the code and our proposed dataset available.