mlsanityData to 13 Oct 2026

Learning Streaming Video Representation via Multitask Training

arXiv 2504.20041 ↗First posted 28 Apr 2025cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Understanding continuous video streams plays a fundamental role in real-time applications including embodied AI and autonomous driving. Unlike offline video understanding, streaming video understanding requires the ability to process video streams frame by frame, preserve historical information, and make low-latency decisions. To address these challenges, our main contributions are three-fold. (i) We develop a novel streaming video backbone, termed as StreamFormer, by incorporating causal temporal attention into a pre-trained vision transformer. This enables efficient streaming video processing while maintaining image representation capability. (ii) To train StreamFormer, we propose to unify diverse spatial-temporal video understanding tasks within a multitask visual-language alignment framework. Hence, StreamFormer learns global semantics, temporal dynamics, and fine-grained spatial relationships simultaneously. (iii) We conduct extensive experiments on online action detection, online video instance segmentation, and video question answering. StreamFormer achieves competitive results while maintaining efficiency, demonstrating its potential for real-time applications.

Key facts. Cited by 10 papers in the map (data through 13 Oct 2026). Builds on 114 of its 135 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

114 of 135 references.

References · 114

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  9. 2023Visual Instruction Tuningcited by 7,827
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,991
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,191
  3. 2023Qwen Technical Reportcited by 3,310
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,264
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,344
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,028
  8. 2018SlowFast Networks for Video Recognitioncited by 1,864
  9. 2021ViViT: A Video Vision Transformercited by 1,615
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,614
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,392
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,389
  13. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,348
  14. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,176
  15. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,150
  16. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  17. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,093
  18. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,092
  19. 2021Video Swin Transformercited by 1,090
  20. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,012
  21. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  22. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 913
  23. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  24. 2021Multiscale Vision Transformerscited by 896
  25. 2016Conditional Image Generation with PixelCNN Decoderscited by 861
  26. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 799
  27. 2017Dense-Captioning Events in Videoscited by 790
  28. 2018Image Transformercited by 771
  29. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 764
  30. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 760
  31. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 744
  32. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 699
  33. 2017Localizing Moments in Video with Natural Languagecited by 661
  34. 2018Taskonomy: Disentangling Task Transfer Learningcited by 633
  35. 2017TALL: Temporal Activity Localization via Language Querycited by 582
  36. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 567
  37. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  38. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 543

50 most cited of 114 All 114 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  2. 2025MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Groundingcited by 11
  3. 2025SoccerMaster: A Vision Foundation Model for Soccer Understandingcited by 10
  4. 2026OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streamscited by 4
  5. 2025Unique Lives, Shared World: Learning from Single-Life Videoscited by 2
  6. 2026ShallowStream: Index Shallow then Answer Deep for Streaming Video Understandingcited by 2
  7. 2026VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimizationcited by 1
  8. 2026StreamSoccer: Event-Driven Memory for Streaming Soccer Commentarycited by 0
  9. 2026MoTE: Mixture of Task Experts for Multi-Task Video Understandingcited by 0
  10. 2026ReactVAU: A Slow-Fast Decoupled Framework for Streaming Video Anomaly Understandingcited by 0