mlsanityData to 7 Oct 2026

Language Repository for Long Video Understanding

arXiv 2403.14622 ↗First posted 21 Mar 2024cs.CVcited by 55

Not on the map. Why?

Abstract (arXiv)

Language has become a prominent modality in computer vision with the rise of LLMs. Despite supporting long context-lengths, their effectiveness in handling long-term information gradually declines with input length. This becomes critical, especially in applications such as long-form video understanding. In this paper, we introduce a Language Repository (LangRepo) for LLMs, that maintains concise and structured information as an interpretable (i.e., all-textual) representation. Our repository is updated iteratively based on multi-scale video chunks. We introduce write and read operations that focus on pruning redundancies in text, and extracting information at various temporal scales. The proposed framework is evaluated on zero-shot visual question-answering benchmarks including EgoSchema, NExT-QA, IntentQA and NExT-GQA, showing state-of-the-art performance at its scale. Our code is available at https://github.com/kkahatapitiya/LangRepo.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 98 of its 120 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

98 of 120 references.

References · 98

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Mistral 7Bcited by 3,065
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2017Non-local Neural Networkscited by 2,290
  9. 2018SlowFast Networks for Video Recognitioncited by 1,862
  10. 2024Mixtral of Expertscited by 1,785
  11. 2021ViViT: A Video Vision Transformercited by 1,610
  12. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  13. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  14. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  15. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  16. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  17. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  18. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  19. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  20. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  21. 2022Token Merging: Your ViT But Fastercited by 995
  22. 2021Multiscale Vision Transformerscited by 896
  23. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  24. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  25. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  26. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  27. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  28. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  29. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  30. 2014Neural Turing Machinescited by 630
  31. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  32. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  33. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  34. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  35. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  36. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  37. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  38. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 26; 2026: 11.

55 papers.

Sort:

By area:

  1. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  2. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  3. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  4. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  5. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  6. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  7. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  8. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  9. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  10. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  11. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  12. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
38 more
  1. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  2. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  3. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Framescited by 27
  4. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26
  5. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  6. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
  7. 2024On the Consistency of Video Large Language Models in Temporal Comprehensioncited by 18
  8. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  9. 2024Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videoscited by 12
  10. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  11. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agentscited by 11
  12. 2026SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLMcited by 11
  13. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  14. 2025Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenderscited by 10
  15. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  16. 2025Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMscited by 9
  17. 2025VideoExpert: Augmented LLM for Temporal-Sensitive Video Understandingcited by 8
  18. 2026Symphony: A Cognitively-Inspired Multi-Agent System for Long-Video Understandingcited by 8
  19. 2024LLAVIDAL: A Large LAnguage VIsion Model for Daily Activities of Livingcited by 7
  20. 2025Fine-Grained Captioning of Long Videos through Scene Graph Consolidationcited by 7
  21. 2025VLog: Video-Language Models by Generative Retrieval of Narration Vocabularycited by 7
  22. 2025AViLA: Asynchronous Vision-Language Agent for Streaming Multimodal Data Interactioncited by 6
  23. 2024FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filteringcited by 5
  24. 2025ReasVQA: Advancing VideoQA with Imperfect Reasoning Processcited by 5
  25. 2025Moment Sampling in Video LLMs for Long-Form Video QAcited by 5
  26. 2025FineQuest: Adaptive Knowledge-Assisted Sports Video Understanding via Agent-of-Thoughts Reasoningcited by 5
  27. 2025MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understandingcited by 4
  28. 2025MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answeringcited by 4
  29. 2025LeAdQA: LLM-Driven Context-Aware Temporal Grounding for Video Question Answeringcited by 4
  30. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4
  31. 2026Soap2Soap: Long Cinematic Video Remaking via Multi-Agent Collaborationcited by 4
  32. 2024VidCtx: Context-aware Video Question Answering with Image Modelscited by 2
  33. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  34. 2025Diffusion-grounded VideoLLM for Entity-aware temoporal groundingcited by 2
  35. 2026Progressive Video Condensation with MLLM Agent for Long-form Video Understandingcited by 2
  36. 2025Detector-Empowered Video Large Language Model for Efficient Spatio-Temporal Groundingcited by 1
  37. 2026Chain-of-Glimpse: Search-Guided Progressive Object-Grounded Reasoning for Video Understandingcited by 1
  38. 2026Why Do Vision Language Models Struggle To Recognize Human Emotions?cited by 1

50 most cited of 55 All 55 (JSON)