mlsanityData to 14 Oct 2026

LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos

arXiv 2312.05269 ↗First posted 7 Dec 2023cs.CV cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

In this paper we introduce LifelongMemory, a new framework for accessing long-form egocentric videographic memory through natural language question answering and retrieval. LifelongMemory generates concise video activity descriptions of the camera wearer and leverages the zero-shot capabilities of pretrained large language models to perform reasoning over long-form video context. Furthermore, LifelongMemory uses a confidence and explanation module to produce confident, high-quality, and interpretable answers. Our approach achieves state-of-the-art performance on the EgoSchema benchmark for question answering and is highly competitive on the natural language query (NLQ) challenge of Ego4D. Code is available at https://github.com/agentic-learning-ai-lab/lifelong-memory.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 47 of its 58 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

47 of 58 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
35 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2017The Kinetics Human Action Video Datasetcited by 1,909
  3. 2018SlowFast Networks for Video Recognitioncited by 1,867
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  7. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  9. 2023A Survey on Multimodal Large Language Modelscited by 909
  10. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  11. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  12. 2017Localizing Moments in Video with Natural Languagecited by 663
  13. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  14. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  15. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  16. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 285
  17. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  18. 2020Span-based Localizing Network for Natural Language Video Localizationcited by 235
  19. 2022Egocentric Video-Language Pretrainingcited by 232
  20. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  21. 2022Learning Video Representations from Large Language Modelscited by 205
  22. 2023Video Language Planningcited by 162
  23. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  24. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
  25. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  26. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  27. 2022InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challengescited by 48
  28. 2023NaQ: Leveraging Narrations as Queries to Supervise Episodic Memorycited by 46
  29. 2023Chat with the Environment: Interactive Multimodal Perception Using Large Language Modelscited by 43
  30. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  31. 2023Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Modelscited by 27
  32. 2023GroundNLQ @ Ego4D Natural Language Queries Challenge 2023cited by 20
  33. 2022ReLER@ZJU-Alibaba Submission to the Ego4D Natural Language Queries Challenge 2022cited by 19
  34. 2023Action Sensitivity Learning for the Ego4D Episodic Memory Challenge 2023cited by 7
  35. 2022Egocentric Video-Language Pretraining @ Ego4D Challenge 2022cited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 22; 2026: 20.

51 papers.

Sort:

By area:

  1. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  2. 2023Video Understanding with Large Language Models: A Surveycited by 206
  3. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  4. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  5. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 65
  6. 2024VCA: Video Curious Agent for Long Video Understandingcited by 56
  7. 2024Language Repository for Long Video Understandingcited by 55
  8. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  9. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 26
  10. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  11. 2024Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoningcited by 17
  12. 2025VideoMultiAgents: A Multi-Agent Framework for Video Question Answeringcited by 16
38 more
  1. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoTcited by 16
  2. 2025Agentic Keyframe Search for Video Question Answeringcited by 13
  3. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  4. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  5. 2025LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understandingcited by 10
  6. 2025Video-EM: Event-Centric Episodic Memory for Long-Form Video Understandingcited by 10
  7. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  8. 2026VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoningcited by 9
  9. 2025How Can Objects Help Video-Language Understanding?cited by 8
  10. 2026Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understandingcited by 8
  11. 2025AViLA: Asynchronous Vision-Language Agent for Streaming Multimodal Data Interactioncited by 6
  12. 2025Spatio-Temporal LLM: Reasoning about Environments and Actionscited by 6
  13. 2025LAST SToP For Modeling Asynchronous Time Seriescited by 5
  14. 2025MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understandingcited by 5
  15. 2025A Grounded Memory System For Smart Personal Assistantscited by 5
  16. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  17. 2026EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchycited by 4
  18. 2025LAST: LeArning to Think in Space and Time for Generalist Vision-Language Modelscited by 3
  19. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3
  20. 2026Pro2Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural Taskscited by 3
  21. 2026Personal Visual Context Learning in Large Multimodal Modelscited by 3
  22. 2026Visual Agentic Memory: Enabling Online Long Video Understanding via Online Indexing, Hierarchical Memory, and Agentic Retrievalcited by 3
  23. 2025Fire360: A Benchmark for Robust Perception and Episodic Memory in Degraded 360-Degree Firefighting Videoscited by 2
  24. 2026Towards Sparse Video Understanding and Reasoningcited by 2
  25. 2026Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understandingcited by 2
  26. 2025Four Eyes Are Better Than Two: Harnessing the Collaborative Potential of Large Models via Differentiated Thinking and Complementary Ensemblescited by 1
  27. 2025DIVE: Deep-search Iterative Video Exploration A Technical Report for the CVRR Challenge at CVPR 2025cited by 1
  28. 2026Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edgecited by 1
  29. 2026HiCrew: Hierarchical Reasoning for Long-Form Video Understanding via Question-Aware Multi-Agent Collaborationcited by 1
  30. 2026EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoningcited by 1
  31. 2026Benchmarking Visual State Tracking in Multimodal Video Understandingcited by 1
  32. 2025FALCONEye: Finding Answers and Localizing Content in ONE-hour-long videos with multi-modal LLMscited by 0
  33. 2026EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environmentcited by 0
  34. 2026HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoningcited by 0
  35. 2026CoVStream: Edge-Cloud Collaboration for Understanding of Long Video Streamscited by 0
  36. 2026Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extractioncited by 0
  37. 2026ViSAGE: Constructing Self-Correcting Memories for Long-Form Video Understandingcited by 0
  38. 2026VideoLoop: Looped Working Memory Against Semantic Thrashing in Long-Form Video Agentscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos
  2. 2026SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory
  3. 2025Advancing Egocentric Video Question Answering with Multimodal Large Language Models
  4. 2025EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answering
  5. 2026The Right Inference Strategy Is All You Need: Nearly Training-Free Domain-Wise Inference for EgoCross Challenge
  6. 2026Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphs
  7. 2026EgoGraph: Temporal Knowledge Graph for Egocentric Video Understanding
  8. 2025How Far Can Off-the-Shelf Multimodal Large Language Models Go in Online Episodic Memory Question Answering?