mlsanityData to 7 Oct 2026

AMEGO: Active Memory from long EGOcentric videos

arXiv 2409.10917 ↗First posted 17 Sep 2024cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Egocentric videos provide a unique perspective into individuals' daily experiences, yet their unstructured nature presents challenges for perception. In this paper, we introduce AMEGO, a novel approach aimed at enhancing the comprehension of very-long egocentric videos. Inspired by the human's ability to maintain information from a single watching, AMEGO focuses on constructing a self-contained representations from one egocentric video, capturing key locations and object interactions. This representation is semantic-free and facilitates multiple queries without the need to reprocess the entire visual content. Additionally, to evaluate our understanding of very-long egocentric videos, we introduce the new Active Memories Benchmark (AMB), composed of more than 20K of highly challenging visual queries from EPIC-KITCHENS. These queries cover different levels of video reasoning (sequencing, concurrency and temporal grounding) to assess detailed video understanding capabilities. We showcase improved performance of AMEGO on AMB, surpassing other video QA baselines by a substantial margin.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 58 of its 74 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

58 of 74 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  6. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  7. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  8. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  9. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  10. 2020Rescaling Egocentric Visioncited by 529
  11. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  12. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
38 more
  1. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  2. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  3. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  4. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  5. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  6. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  7. 2022FlowFormer: A Transformer Architecture for Optical Flowcited by 273
  8. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  9. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  10. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  11. 2015Structural-RNN: Deep Learning on Spatio-Temporal Graphscited by 234
  12. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  13. 2018Videos as Space-Time Region Graphscited by 221
  14. 2022Learning Video Representations from Large Language Modelscited by 204
  15. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  16. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  17. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  18. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  19. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  20. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  21. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
  22. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  23. 2017Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Rewardcited by 105
  24. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
  25. 2018Actor-Centric Relation Networkcited by 87
  26. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  27. 2018Video Summarization Using Fully Convolutional Sequence Networkscited by 71
  28. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  29. 2018Object Level Visual Reasoning in Videoscited by 67
  30. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  31. 2019VideoGraph: Recognizing Minutes-Long Human Activities in Videoscited by 57
  32. 2023Action Scene Graphs for Long-Form Understanding of Egocentric Videoscited by 51
  33. 2023EPIC Fields: Marrying 3D Geometry and Video Understandingcited by 49
  34. 2023NaQ: Leveraging Narrations as Queries to Supervise Episodic Memorycited by 46
  35. 2021Unified Graph Structured Models for Video Understandingcited by 38
  36. 2017Attend and Interact: Higher-Order Object Interactions for Video Understandingcited by 37
  37. 2022Episodic Memory Question Answeringcited by 32
  38. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 19; 2026: 14.

35 papers.

Sort:

By area:

  1. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  2. 2025MemER: Scaling Up Memory for Robot Control via Experience Retrievalcited by 64
  3. 2025ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videoscited by 21
  4. 2025In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Promptingcited by 21
  5. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
  6. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  7. 2025Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMscited by 19
  8. 2025The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimationcited by 16
  9. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  10. 2024Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memorycited by 11
  11. 2026EgoSound: Benchmarking Sound Understanding in Egocentric Videoscited by 8
  12. 2025HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videoscited by 7
23 more
  1. 2026Ego-Grounding for Personalized Question-Answering in Egocentric Videoscited by 6
  2. 2025Hier-EgoPack: Hierarchical Egocentric Video Understanding with Diverse Task Perspectivescited by 5
  3. 2025A Grounded Memory System For Smart Personal Assistantscited by 5
  4. 2025EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphscited by 5
  5. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  6. 2024Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experiencecited by 4
  7. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4
  8. 2026VisionClaw: Always-On AI Agents through Smart Glassescited by 4
  9. 2026EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Visioncited by 3
  10. 2025HERO-VQL: Hierarchical, Egocentric and Robust Visual Query Localizationcited by 2
  11. 2025HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamicscited by 2
  12. 2026EgoSelf: From Memory to Personalized Egocentric Assistantcited by 2
  13. 2026Whareformer: Learning to Track What is Where in Long Egocentric Videoscited by 2
  14. 2025Is Tracking really more challenging in First Person Egocentric Vision?cited by 1
  15. 2025Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answeringcited by 1
  16. 2026S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrievalcited by 1
  17. 2026R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Videocited by 1
  18. 2026EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environmentcited by 0
  19. 2026EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memorycited by 0
  20. 2026Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agentscited by 0
  21. 2026AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalizationcited by 0
  22. 2026Does Local Video Understanding Transfer Across Encounters? The EgoGears Benchmarkcited by 0
  23. 2026MemLife: Curating and Reasoning over Long-Term Egocentric Video Memoriescited by 0