# LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation

arXiv: https://arxiv.org/abs/2306.08736 · HTML: https://mlsanity.com/paper/2306.08736/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 59 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [A Simple Method for Commonsense Reasoning](https://mlsanity.com/paper/1806.02847/) (2018)
- [Language as Queries for Referring Video Object Segmentation](https://mlsanity.com/paper/2201.00487/) (2022)
- [Video Object Segmentation with Language Referring Expressions](https://mlsanity.com/paper/1803.08006/) (2018)
- [End-to-End Referring Video Object Segmentation with Multimodal Transformers](https://mlsanity.com/paper/2111.14821/) (2021)
- [Actor and Action Video Segmentation from a Sentence](https://mlsanity.com/paper/1803.07485/) (2018)
- [Cross-Modal Progressive Comprehension for Referring Segmentation](https://mlsanity.com/paper/2105.07175/) (2021)
- [Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition](https://mlsanity.com/paper/1711.11152/) (2017)
- [Semantic Video Segmentation by Gated Recurrent Flow Propagation](https://mlsanity.com/paper/1612.08871/) (2016)
- [SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation](https://mlsanity.com/paper/2305.17011/) (2023)
- [On the Integration of Optical Flow and Action Recognition](https://mlsanity.com/paper/1712.08416/) (2017)
- [Rethinking Cross-modal Interaction from a Top-down Perspective for Referring Video Object Segmentation](https://mlsanity.com/paper/2106.01061/) (2021)

## Built on by (most cited first, 23 of 23)

- [MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation](https://mlsanity.com/paper/2512.10945/) (2025)
- [ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations](https://mlsanity.com/paper/2501.14607/) (2025)
- [Text Promptable Surgical Instrument Segmentation with Vision-Language Models](https://mlsanity.com/paper/2306.09244/) (2023)
- [Bootstrapping Referring Multi-Object Tracking](https://mlsanity.com/paper/2406.05039/) (2024)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation](https://mlsanity.com/paper/2507.22886/) (2025)
- [Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation](https://mlsanity.com/paper/2602.03595/) (2026)
- [MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation](https://mlsanity.com/paper/2501.13667/) (2025)
- [Towards High-Consistency Embodied World Model with Multi-View Trajectory Videos](https://mlsanity.com/paper/2511.12882/) (2025)
- [Implicit Counterfactual Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2507.20740/) (2025)
- [MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understanding](https://mlsanity.com/paper/2510.09274/) (2025)
- [Learning Temporal 3D Semantic Scene Completion via Optical Flow Guidance](https://mlsanity.com/paper/2502.14520/) (2025)
- [Find First, Track Next: Decoupling Identification and Propagation in Referring Video Object Segmentation](https://mlsanity.com/paper/2503.03492/) (2025)
- [GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation](https://mlsanity.com/paper/2406.12834/) (2024)
- [VRMDiff: Text-Guided Video Referring Matting Generation of Diffusion](https://mlsanity.com/paper/2503.10678/) (2025)
- [RefAM: Attention Magnets for Zero-Shot Referral Segmentation](https://mlsanity.com/paper/2509.22650/) (2025)
- [Harnessing Vision-Language Pretrained Models with Temporal-Aware Adaptation for Referring Video Object Segmentation](https://mlsanity.com/paper/2405.10610/) (2024)
- [ReferDINO-Plus: 2nd Solution for 4th PVUW MeViS Challenge at CVPR 2025](https://mlsanity.com/paper/2503.23509/) (2025)
- [AR2-4FV: Anchored Referring and Re-identification for Long-Term Grounding in Fixed-View Videos](https://mlsanity.com/paper/2603.07758/) (2026)
- [Temporal Prompting Matters: Rethinking Referring Video Object Segmentation](https://mlsanity.com/paper/2510.07319/) (2025)
- [Weakly-Supervised Referring Video Object Segmentation through Text Supervision](https://mlsanity.com/paper/2604.17797/) (2026)
- [Event-Aware Instructed Assistant for Referring Video Segmentation](https://mlsanity.com/paper/2606.26994/) (2026)
- [LD-RSVIS: A Large-Scale and Diverse Benchmark for Referring Surgical Video Instrument Segmentation](https://mlsanity.com/paper/2609.23067/) (2026)

## Abstract (arXiv)

Referring video object segmentation (RVOS) aims to segment the target instance referred by a given text expression in a video clip. The text expression normally contains sophisticated description of the instance's appearance, action, and relation with others. It is therefore rather difficult for a RVOS model to capture all these attributes correspondingly in the video; in fact, the model often favours more on the action- and relation-related visual attributes of the instance. This can end up with partial or even incorrect mask prediction of the target instance. We tackle this problem by taking a subject-centric short text expression from the original long text expression. The short one retains only the appearance-related information of the target instance so that we can use it to focus the model's attention on the instance's appearance. We let the model make joint predictions using both long and short text expressions; and insert a long-short cross-attention module to interact the joint features and a long-short predictions intersection loss to regulate the joint predictions. Besides the improvement on the linguistic part, we also introduce a forward-backward visual consistency loss, which utilizes optical flows to warp visual features between the annotated frames and their temporal neighbors for consistency. We build our method on top of two state of the art pipelines. Extensive experiments on A2D-Sentences, Refer-YouTube-VOS, JHMDB-Sentences and Refer-DAVIS17 show impressive improvements of our method.Code is available at https://github.com/LinfengYuan1997/Losh.