mlsanityData to 14 Oct 2026

Actor and Action Video Segmentation from a Sentence

arXiv 1803.07485 ↗First posted 20 Mar 2018cs.CVcited by 106

Not on the map. Why?

Abstract (arXiv)

This paper strives for pixel-level segmentation of actors and their actions in video content. Different from existing works, which all learn to segment from a fixed vocabulary of actor and action pairs, we infer the segmentation from a natural language input sentence. This allows to distinguish between fine-grained actors in the same super-category, identify actor and action instances, and segment pairs that are outside of the actor and action vocabulary. We propose a fully-convolutional model for pixel-level actor and action segmentation using an encoder-decoder architecture optimized for video. To show the potential of actor and action video segmentation from a sentence, we extend two popular actor and action datasets with more than 7,500 natural language descriptions. Experiments demonstrate the quality of the sentence-guided segmentations, the generalization ability of our model, and its advantage for traditional actor and action segmentation compared to the state-of-the-art.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 7; 2023: 17; 2024: 26; 2025: 24; 2026: 19.

106 papers.

Sort:

By area:

  1. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  2. 2022Language as Queries for Referring Video Object Segmentationcited by 182
  3. 2021End-to-End Referring Video Object Segmentation with Multimodal Transformerscited by 135
  4. 2021A Survey on Deep Learning Technique for Video Segmentationcited by 132
  5. 2021Cross-Modal Progressive Comprehension for Referring Segmentationcited by 102
  6. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 100
  7. 2023Referring Multi-Object Trackingcited by 93
  8. 2023OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentationcited by 62
  9. 2023SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentationcited by 57
  10. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  11. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 51
  12. 2022Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentationcited by 50
38 more
  1. 2023Spectrum-guided Multi-granularity Referring Video Object Segmentationcited by 48
  2. 2024Elysium: Exploring Object-level Perception in Videos via MLLMcited by 48
  3. 2024LVOS: A Benchmark for Large-scale Long-term Video Object Segmentationcited by 44
  4. 2024SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentationcited by 44
  5. 2021Rethinking Cross-modal Interaction from a Top-down Perspective for Referring Video Object Segmentationcited by 42
  6. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  7. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 35
  8. 2024Towards Language-Driven Video Inpainting via Multimodal Large Language Modelscited by 33
  9. 2024Ref-AVS: Refer and Segment Objects in Audio-Visual Scenescited by 33
  10. 2021Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentationcited by 32
  11. 2025ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundationscited by 30
  12. 2021Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Groundingcited by 29
  13. 2021Referring Segmentation in Images and Videos with Cross-Modal Self-Attention Networkcited by 28
  14. 2022Multi-Attention Network for Compressed Video Referring Object Segmentationcited by 28
  15. 2023Temporal Collection and Distribution for Referring Video Object Segmentationcited by 27
  16. 2024Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentationcited by 27
  17. 2023RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentationcited by 25
  18. 2024Artemis: Towards Referential Understanding in Complex Videoscited by 25
  19. 2023LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentationcited by 23
  20. 2024Bootstrapping Referring Multi-Object Trackingcited by 22
  21. 2025Multimodal Referring Segmentation: A Surveycited by 22
  22. 2021ClawCraneNet: Leveraging Object-level Relation for Text-based Video Segmentationcited by 21
  23. 2024EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Drivingcited by 19
  24. 2022Modeling Motion with Multi-Modal Features for Text-Based Video Segmentationcited by 18
  25. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 17
  26. 2023Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentationcited by 16
  27. 2024Referring Atomic Video Action Recognitioncited by 16
  28. 2024Temporally Consistent Referring Video Object Segmentation with Hybrid Memorycited by 14
  29. 2025Fine-grained Spatiotemporal Grounding on Egocentric Videoscited by 11
  30. 2024ReferEverything: Towards Segmenting Everything We Can Speak of in Videoscited by 10
  31. 2025Deforming Videos to Masks: Flow Matching for Referring Video Segmentationcited by 10
  32. 2023Adversarial Attacks on Video Object Segmentation with Hard Region Discoverycited by 9
  33. 2024ActionVOS: Actions as Prompts for Video Object Segmentationcited by 8
  34. 2023Learning Referring Video Object Segmentation from Weak Annotationcited by 6
  35. 2023Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samplescited by 6
  36. 2024SafaRi:Adaptive Sequence Transformer for Weakly Supervised Referring Expression Segmentationcited by 6
  37. 2024Temporal-Enhanced Multimodal Transformer for Referring Multi-Object Tracking and Segmentationcited by 6
  38. 2025InterRVOS: Interaction-aware Referring Video Object Segmentationcited by 6

50 most cited of 106 All 106 (JSON)