mlsanityData to 14 Oct 2026

Learning Space-Time Semantic Correspondences

arXiv 2306.10208 ↗First posted 16 Jun 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose a new task of space-time semantic correspondence prediction in videos. Given a source video, a target video, and a set of space-time key-points in the source video, the task requires predicting a set of keypoints in the target video that are the semantic correspondences of the provided source keypoints. We believe that this task is important for fine-grain video understanding, potentially enabling applications such as activity coaching, sports analysis, robot imitation learning, and more. Our contributions in this paper are: (i) proposing a new task and providing annotations for space-time semantic correspondences on two existing benchmarks: Penn Action and Pouring; and (ii) presenting a comprehensive set of baselines and experiments to gain insights about the new problem. Our main finding is that the space-time semantic correspondence prediction problem is best approached jointly in space and time rather than in their decomposed sub-problems: time alignment and spatial correspondences.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

27 of 38 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017The Kinetics Human Action Video Datasetcited by 1,909
  3. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  4. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  5. 2017Soft-DTW: a Differentiable Loss Function for Time-Seriescited by 271
  6. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  7. 2018Neighbourhood Consensus Networkscited by 174
  8. 2019Temporal Cycle-Consistency Learningcited by 154
  9. 2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 146
  10. 2019Few-Shot Video Classification via Temporal Alignmentcited by 125
  11. 2022Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentationcited by 114
  12. 2019D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentationcited by 83
15 more
  1. 2016Universal Correspondence Networkcited by 82
  2. 2021CATs: Cost Aggregation Transformers for Visual Correspondencecited by 77
  3. 2015Unsupervised Object Discovery and Localization in the Wild: Part-based Matching with Bottom-up Region Proposalscited by 69
  4. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 65
  5. 2019Hyperpixel Flow: Semantic Correspondence with Multi-layer Neural Featurescited by 56
  6. 2020Learning to Compose Hypercolumns for Visual Correspondencecited by 53
  7. 2022CATs++: Boosting Cost Aggregation with Convolutions and Transformerscited by 53
  8. 2017SCNet: Learning Semantic Correspondencecited by 48
  9. 2021Learning by Aligning Videos in Timecited by 48
  10. 2017Proposal Flow: Semantic Correspondences from Object Proposalscited by 47
  11. 2021Convolutional Hough Matching Networkscited by 43
  12. 2017FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondencecited by 42
  13. 2014Do Convnets Learn Correspondence?cited by 28
  14. 2015Dense Semantic Correspondence where Every Pixel is a Classifiercited by 15
  15. 2017DCTM: Discrete-Continuous Transformation Matching for Semantic Flowcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024StarVid: Enhancing Semantic Alignment in Video Diffusion Models via Spatial and SynTactic Guided Attention Refocusing
  2. 2025From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approach
  3. 2025Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoning
  4. 2026TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs
  5. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model
  6. 2026MARCO: Navigating the Unseen Space of Semantic Correspondence
  7. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data
  8. 2024FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation