mlsanityData to 7 Oct 2026

LITA: Language Instructed Temporal-Localization Assistant

arXiv 2403.19046 ↗First posted 27 Mar 2024cs.CV cs.AIcited by 124

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

There has been tremendous progress in multimodal Large Language Models (LLMs). Recent works have extended these models to video input with promising instruction following capabilities. However, an important missing piece is temporal localization. These models cannot accurately answer the "When?" questions. We identify three key aspects that limit their temporal localization capabilities: (i) time representation, (ii) architecture, and (iii) data. We address these shortcomings by proposing Language Instructed Temporal-Localization Assistant (LITA) with the following features: (1) We introduce time tokens that encode timestamps relative to the video length to better represent time in videos. (2) We introduce SlowFast tokens in the architecture to capture temporal information at fine temporal resolution. (3) We emphasize temporal localization data for LITA. In addition to leveraging existing video datasets with timestamps, we propose a new task, Reasoning Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for learning and evaluating this task. Reasoning temporal localization requires both the reasoning and temporal localization of Video LLMs. LITA demonstrates strong performance on this challenging task, nearly doubling the temporal mean intersection-over-union (mIoU) of baselines. In addition, we show that our emphasis on temporal localization also substantially improves video-based text generation compared to existing Video LLMs, including a 36% relative improvement of Temporal Understanding. Code is available at: https://github.com/NVlabs/LITA

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 43 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

40 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  2. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun
  5. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning27 Feb

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2018SlowFast Networks for Video Recognitioncited by 1,862
23 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  5. 2017Dense-Captioning Events in Videoscited by 789
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  7. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  9. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  10. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  11. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  12. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  13. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  14. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  15. 2020Rescaling Egocentric Visioncited by 529
  16. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  17. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  18. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  19. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  20. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  21. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  22. 2020Multimodal Pretraining for Dense Video Captioningcited by 79
  23. 2023UnLoc: A Unified Framework for Video Localization Taskscited by 66

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 32; 2025: 61; 2026: 31.

124 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  3. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  4. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  5. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  6. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  7. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  8. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  9. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  10. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
  11. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  12. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
38 more
  1. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  2. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  3. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  4. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  5. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  6. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  7. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 58
  8. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  9. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 52
  10. 2024ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videoscited by 51
  11. 2024X-VILA: Cross-Modality Alignment for Large Language Modelcited by 46
  12. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  13. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  14. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
  15. 20253D Aware Region Prompted Vision Language Modelcited by 31
  16. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  17. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  18. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  19. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  20. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Modelscited by 26
  21. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiencycited by 25
  22. 2024Neptune: The Long Orbit to Benchmarking Long Video Understandingcited by 24
  23. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 23
  24. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  25. 2024Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Levelcited by 18
  26. 2024On the Consistency of Video Large Language Models in Temporal Comprehensioncited by 18
  27. 2024ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videoscited by 18
  28. 2025Datasets and Recipes for Video Temporal Grounding via Reinforcement Learningcited by 18
  29. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 18
  30. 2024Chrono: A Simple Blueprint for Representing Time in MLLMscited by 17
  31. 2024TimeRefine: Temporal Grounding with Time Refining Video LLMcited by 15
  32. 2024HandsOnVLM: Vision-Language Models for Hand-Object Interaction Predictioncited by 15
  33. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  34. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generationcited by 13
  35. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 13
  36. 2024Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Groundingcited by 12
  37. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  38. 2025Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Visioncited by 11

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Temporal Grounding of Activities using Multimodal Large Language Models
  2. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability
  3. 2025Chain-of-Evidence Multimodal Reasoning for Few-shot Temporal Action Localization
  4. 2025DaMO: A Data-Efficient Multimodal Orchestrator for Temporal Reasoning with Video LLMs
  5. 2024Do Language Models Understand Time?
  6. 2024Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models
  7. 2025Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models
  8. 2025TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learning