mlsanityData to 7 Oct 2026

Is an Object-Centric Video Representation Beneficial for Transfer?

arXiv 2207.10075 ↗First posted 20 Jul 2022cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

The objective of this work is to learn an object-centric video representation, with the aim of improving transferability to novel tasks, i.e., tasks different from the pre-training task of action classification. To this end, we introduce a new object-centric video recognition model based on a transformer architecture. The model learns a set of object-centric summary vectors for the video, and uses these vectors to fuse the visual and spatio-temporal trajectory 'modalities' of the video clip. We also introduce a novel trajectory contrast loss to further enhance objectness in these summary vectors. With experiments on four datasets -- SomethingSomething-V2, SomethingElse, Action Genome and EpicKitchens -- we show that the object-centric model outperforms prior video representations (both object-agnostic and object-aware), when: (1) classifying actions on unseen objects and unseen environments; (2) low-shot learning of novel classes; (3) linear probe to other downstream tasks; as well as (4) for standard action classification.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 79 of its 98 references that are in the map. First posted on arXiv 20 Jul 2022.

Builds on

79 of 98 references.

References · 79

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017Mask R-CNNcited by 7,259
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2017Non-local Neural Networkscited by 2,290
  11. 2018SlowFast Networks for Video Recognitioncited by 1,862
  12. 2021ViViT: A Video Vision Transformercited by 1,610
38 more
  1. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  2. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  3. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  4. 2016Deep Networks with Stochastic Depthcited by 973
  5. 2020Object-Centric Learning with Slot Attentioncited by 865
  6. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  7. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
  8. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  9. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  10. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  11. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  12. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  13. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  14. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
  15. 2019Video Instance Segmentationcited by 391
  16. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  17. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  18. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  19. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  20. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  21. 2018Image Generation from Scene Graphscited by 324
  22. 2018Video Action Transformer Networkcited by 316
  23. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  24. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  25. 2017Objects that Soundcited by 254
  26. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  27. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  28. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  29. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  30. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  31. 2018Videos as Space-Time Region Graphscited by 221
  32. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  33. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  34. 2020Temporal Pyramid Network for Action Recognitioncited by 163
  35. 2021Self-supervised Video Object Segmentation by Motion Groupingcited by 147
  36. 2020TAO: A Large-Scale Benchmark for Tracking Any Objectcited by 146
  37. 2019Unsupervised State Representation Learning in Ataricited by 144
  38. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 11; 2024: 5; 2025: 5; 2026: 1.

24 papers.

Sort:

By area:

  1. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 85
  2. 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 36
  3. 2023Helping Hands: An Object-Aware Ego-Centric Video Recognition Modelcited by 31
  4. 2023Multimodal Distillation for Egocentric Action Recognitioncited by 28
  5. 2023Object-centric Video Representation for Long-term Action Anticipationcited by 24
  6. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  7. 2023How can objects help action recognition?cited by 21
  8. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 18
  9. 2023Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoningcited by 14
  10. 2025PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planningcited by 13
  11. 2024GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognitioncited by 8
  12. 2024Simultaneous Detection and Interaction Reasoning for Object-Centric Action Recognitioncited by 8
12 more
  1. 2024Rethinking Image-to-Video Adaptation: An Object-centric Perspectivecited by 8
  2. 2024Reasoning-Enhanced Object-Centric Learning for Videoscited by 7
  3. 2023RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in Object-centric Learningcited by 6
  4. 2023ST(OR)2: Spatio-Temporal Object Level Reasoning for Activity Recognition in the Operating Roomcited by 6
  5. 2025Object-Centric Representations Improve Policy Generalization in Robot Manipulationcited by 6
  6. 2025EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perceptioncited by 4
  7. 2023MOFO: MOtion FOcused Self-Supervision for Video Understandingcited by 3
  8. 2025Is an object-centric representation beneficial for robotic manipulation ?cited by 3
  9. 2022Students taught by multimodal teachers are superior action recognizerscited by 2
  10. 2023Modelling Spatio-Temporal Interactions For Compositional Action Recognitioncited by 1
  11. 2023Does Visual Pretraining Help End-to-End Reasoning?cited by 1
  12. 2026DAOS: A Multimodal In-cabin Behavior Monitoring with Driver Action-Object Synergy Datasetcited by 1