# SEAL: Semantic Attention Learning for Long Video Representation

arXiv: https://arxiv.org/abs/2412.01798 · HTML: https://mlsanity.com/paper/2412.01798/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 45 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Builds on (most cited first, 30 of 38)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [YOLOv10: Real-Time End-to-End Object Detection](https://mlsanity.com/paper/2405.14458/) (2024)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [BoT-SORT: Robust Associations Multi-Pedestrian Tracking](https://mlsanity.com/paper/2206.14651/) (2022)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024)
- [MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding](https://mlsanity.com/paper/2404.05726/) (2024)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Streaming Long Video Understanding with Large Language Models](https://mlsanity.com/paper/2405.16009/) (2024)
- [Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution](https://mlsanity.com/paper/2409.12961/) (2024)
- [LongVLM: Efficient Long Video Understanding via Large Language Models](https://mlsanity.com/paper/2404.03384/) (2024)
- [MoReVQA: Exploring Modular Reasoning Models for Video Question Answering](https://mlsanity.com/paper/2404.06511/) (2024)
- [MovieChat+: Question-aware Sparse Memory for Long Video Question Answering](https://mlsanity.com/paper/2404.17176/) (2024)
- [Koala: Key frame-conditioned long video-LLM](https://mlsanity.com/paper/2404.04346/) (2024)
- [From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answering](https://mlsanity.com/paper/2205.14895/) (2022)
- [EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation](https://mlsanity.com/paper/2406.18070/) (2024)

## Built on by (most cited first, 13 of 13)

- [Training-Free Multi-Style Fusion Through Reference-Based Adaptive Modulation](https://mlsanity.com/paper/2509.18602/) (2025)
- [Failures to Surface Harmful Contents in Video Large Language Models](https://mlsanity.com/paper/2508.10974/) (2025)
- [Prompt-Guided Dual Latent Steering for Inversion Problems](https://mlsanity.com/paper/2509.18619/) (2025)
- [SeViCES: Unifying Semantic-Visual Evidence Consensus for Long Video Understanding](https://mlsanity.com/paper/2510.20622/) (2025)
- [Poisoning Prompt-Guided Sampling in Video Large Language Models](https://mlsanity.com/paper/2509.20851/) (2025)
- [Enhancing Long Video Question Answering with Scene-Localized Frame Grouping](https://mlsanity.com/paper/2508.03009/) (2025)
- [AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding](https://mlsanity.com/paper/2603.28696/) (2026)
- [SAVA-X: Ego-to-Exo Imitation Error Detection via Scene-Adaptive View Alignment and Bidirectional Cross View Fusion](https://mlsanity.com/paper/2603.12764/) (2026)
- [CoVStream: Edge-Cloud Collaboration for Understanding of Long Video Streams](https://mlsanity.com/paper/2606.22804/) (2026)
- [QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding](https://mlsanity.com/paper/2607.04559/) (2026)
- [Do Input-Level Defenses Transfer to Observation-Level Attacks on VideoLLMs?](https://mlsanity.com/paper/2609.08331/) (2026)
- [dKFD: Phase-Structured Evidence Allocation for Fixed-Budget Localized Event Understanding](https://mlsanity.com/paper/2609.33083/) (2026)
- [VideoJev: System One Model for Video Understanding and Interaction](https://mlsanity.com/paper/2610.18615/) (2026)

## Abstract (arXiv)

Long video understanding presents challenges due to the inherent high computational complexity and redundant temporal information. An effective representation for long videos must efficiently process such redundancy while preserving essential contents for downstream tasks. This paper introduces SEmantic Attention Learning (SEAL), a novel unified representation for long videos. To reduce computational complexity, long videos are decomposed into three distinct types of semantic entities: scenes, objects, and actions, allowing models to operate on a compact set of entities rather than a large number of frames or pixels. To further address redundancy, we propose an attention learning module that balances token relevance with diversity, formulated as a subset selection optimization problem. Our representation is versatile and applicable across various long video understanding tasks. Extensive experiments demonstrate that SEAL significantly outperforms state-of-the-art methods in video question answering and temporal grounding tasks across diverse benchmarks, including LVBench, MovieChat-1K, and Ego4D.