mlsanityData to 14 Oct 2026

Exploring High-Order Self-Similarity for Video Understanding

arXiv 2604.20760 ↗First posted 22 Apr 2026cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Space-time self-similarity (STSS), which captures visual correspondences across frames, provides an effective way to represent temporal dynamics for video understanding. In this work, we explore higher-order STSS and demonstrate how STSSs at different orders reveal distinct aspects of these dynamics. We then introduce the Multi-Order Self-Similarity (MOSS) module, a lightweight neural module designed to learn and integrate multi-order STSS features. It can be applied to diverse video tasks to enhance motion modeling capabilities while consuming only marginal computational cost and memory usage. Extensive experiments on video action recognition, motion-centric video VQA, and real-world robotic tasks consistently demonstrate substantial improvements, validating the broad applicability of MOSS as a general temporal modeling module. The source code and checkpoints will be publicly available.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 84 of its 100 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

84 of 100 references.

References · 84

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2018SlowFast Networks for Video Recognitioncited by 1,867
  11. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  12. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
38 more
  1. 2021ViViT: A Video Vision Transformercited by 1,620
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  5. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  7. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  9. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  12. 2021Video Swin Transformercited by 1,091
  13. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  14. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  15. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  16. 2021Multiscale Vision Transformerscited by 897
  17. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  18. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  19. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  20. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  21. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
  22. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  23. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  24. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  25. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  26. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  27. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  28. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  29. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  30. 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 234
  31. 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
  32. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  33. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 193
  34. 2022Frozen CLIP Models are Efficient Video Learnerscited by 182
  35. 2022Multiview Transformers for Video Recognitioncited by 148
  36. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 148
  37. 2019Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognitioncited by 117
  38. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026RLDX-1 Technical Reportcited by 28