mlsanityData to 7 Oct 2026

SViTT-Ego: A Sparse Video-Text Transformer for Egocentric Video

arXiv 2406.09462 ↗First posted 13 Jun 2024cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Pretraining egocentric vision-language models has become essential to improving downstream egocentric video-text tasks. These egocentric foundation models commonly use the transformer architecture. The memory footprint of these models during pretraining can be substantial. Therefore, we pretrain SViTT-Ego, the first sparse egocentric video-text transformer model integrating edge and node sparsification. We pretrain on the EgoClip dataset and incorporate the egocentric-friendly objective EgoNCE, instead of the frequently used InfoNCE. Most notably, SViTT-Ego obtains a +2.8% gain on EgoMCQ (intra-video) accuracy compared to LAVILA large, with no additional data augmentation techniques other than standard image augmentations, yet pretrainable on memory-limited devices.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 18 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

11 of 18 references.

References · 11

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2022Token Merging: Your ViT But Fastercited by 995
  5. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  6. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  7. 2020Span-based Localizing Network for Natural Language Video Localizationcited by 234
  8. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  9. 2022Learning Video Representations from Large Language Modelscited by 204
  10. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  11. 2023SViTT: Temporal Learning of Sparse Video-Text Transformerscited by 11

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interactioncited by 5
  2. 2025EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimizationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
  2. 2025EgoX: Egocentric Video Generation from a Single Exocentric Video
  3. 2024Retrieval-Augmented Egocentric Video Captioning
  4. 2025From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision
  5. 2024EAGLE: Egocentric AGgregated Language-video Engine
  6. 2024X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization
  7. 2026EgoAVU: Egocentric Audio-Visual Understanding
  8. 2026LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension