# Context-Aware Temporal Embedding of Objects in Video Data

arXiv: https://arxiv.org/abs/2408.12789 · HTML: https://mlsanity.com/paper/2408.12789/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 56 references that are in the map.
- First posted on arXiv 23 Aug 2024.

## Builds on (most cited first, 5 of 5)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Selfie: Self-supervised Pretraining for Image Embedding](https://mlsanity.com/paper/1906.02940/) (2019)

## Abstract (arXiv)

In video analysis, understanding the temporal context is crucial for recognizing object interactions, event patterns, and contextual changes over time. The proposed model leverages adjacency and semantic similarities between objects from neighboring video frames to construct context-aware temporal object embeddings. Unlike traditional methods that rely solely on visual appearance, our temporal embedding model considers the contextual relationships between objects, creating a meaningful embedding space where temporally connected object's vectors are positioned in proximity. Empirical studies demonstrate that our context-aware temporal embeddings can be used in conjunction with conventional visual embeddings to enhance the effectiveness of downstream applications. Moreover, the embeddings can be used to narrate a video using a Large Language Model (LLM). This paper describes the intricate details of the proposed objective function to generate context-aware temporal object embeddings for video data and showcases the potential applications of the generated embeddings in video analysis and object classification tasks.