mlsanityData to 7 Oct 2026

Prototypical Transformer as Unified Motion Learners

arXiv 2406.01559 ↗First posted 3 Jun 2024cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

In this work, we introduce the Prototypical Transformer (ProtoFormer), a general and unified framework that approaches various motion tasks from a prototype perspective. ProtoFormer seamlessly integrates prototype learning with Transformer by thoughtfully considering motion dynamics, introducing two innovative designs. First, Cross-Attention Prototyping discovers prototypes based on signature motion patterns, providing transparency in understanding motion scenes. Second, Latent Synchronization guides feature representation learning via prototypes, effectively mitigating the problem of motion uncertainty. Empirical results demonstrate that our approach achieves competitive performance on popular motion tasks such as optical flow and scene depth. Furthermore, it exhibits generality across various downstream tasks, including object tracking and video stabilization.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 85 of its 132 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

85 of 132 references.

References · 85

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  9. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  10. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  12. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
38 more
  1. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  2. 2021Transformers in Vision: A Surveycited by 1,009
  3. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  4. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  5. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  6. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  7. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  8. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  9. 2020A Survey on Visual Transformercited by 653
  10. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  11. 2020AdaBins: Depth Estimation using Adaptive Binscited by 596
  12. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
  13. 2019PANet: Few-Shot Image Semantic Segmentation with Prototype Alignmentcited by 449
  14. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  15. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 424
  16. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 421
  17. 2021Transformer Trackingcited by 412
  18. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  19. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  20. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  21. 2020Virtual KITTI 2cited by 345
  22. 2018TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wildcited by 342
  23. 2018DVC: An End-to-end Deep Video Compression Frameworkcited by 324
  24. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 324
  25. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  26. 2022FlowFormer: A Transformer Architecture for Optical Flowcited by 273
  27. 2016ECO: Efficient Convolution Operators for Trackingcited by 239
  28. 2019Enforcing geometric constraints of virtual normal for depth predictioncited by 233
  29. 2021Learning to Estimate Hidden Motions with Global Motion Aggregationcited by 223
  30. 2021Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Trackingcited by 220
  31. 2020Ocean: Object-aware Anchor-free Trackingcited by 212
  32. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  33. 2015Learning Multi-Domain Convolutional Neural Networks for Visual Trackingcited by 209
  34. 2021Adaptive Prototype Learning and Allocation for Few-Shot Segmentationcited by 207
  35. 2020Prototype Mixture Models for Few-shot Semantic Segmentationcited by 190
  36. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  37. 2020Part-aware Prototype Network for Few-shot Semantic Segmentationcited by 162
  38. 2021FVC: A New Framework towards Deep Video Compression in Feature Spacecited by 149

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024AMD: Automatic Multi-step Distillation of Large-scale Vision Modelscited by 12
  2. 2025Re-Imagining Multimodal Instruction Tuning: A Representation Viewcited by 9
  3. 2025Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modelingcited by 2
  4. 2026XSPLAIN: XAI-enabling Splat-based Prototype Learning for Attribute-aware INterpretabilitycited by 0