mlsanityData to 7 Oct 2026

Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding

arXiv 2502.06020 ↗First posted 9 Feb 2025cs.CV cs.MM cs.SDcited by 28

Not on the map. Why?

Abstract (arXiv)

Multimodal foundation models (MFMs) have demonstrated significant success in tasks such as visual captioning, question answering, and image-text retrieval. However, these models face inherent limitations due to their finite internal capacity, which restricts their ability to process extended temporal sequences, a crucial requirement for comprehensive video and audio analysis. To overcome these challenges, we introduce a specialized cognitive module, temporal working memory (TWM), which aims to enhance the temporal modeling capabilities of MFMs. It selectively retains task-relevant information across temporal dimensions, ensuring that critical details are preserved throughout the processing of video and audio content. The TWM uses a query-guided attention approach to focus on the most informative multimodal segments within temporal sequences. By retaining only the most relevant content, TWM optimizes the use of the model's limited capacity, enhancing its temporal modeling ability. This plug-and-play module can be easily integrated into existing MFMs. With our TWM, nine state-of-the-art models exhibit significant performance improvements across tasks such as video captioning, question answering, and video-text retrieval. By enhancing temporal modeling, TWM extends the capability of MFMs to handle complex, time-sensitive data effectively. Our code is available at https://github.com/xid32/NAACL_2025_TWM.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

31 of 49 references.

References · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  8. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  9. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  10. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  11. 2022Large Language Models with Controllable Working Memorycited by 169
  12. 2021Towards Long-Form Video Understandingcited by 146
19 more
  1. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  2. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  3. 2020Condensed Movies: Story Based Retrieval with Contextual Embeddingscited by 95
  4. 2018Timeception for Complex Action Recognitioncited by 89
  5. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  6. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  7. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Benchcited by 61
  8. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  9. 2024InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoningcited by 36
  10. 2023Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Taskscited by 31
  11. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  12. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 28
  13. 2023Working Memory Capacity of ChatGPT: An Empirical Studycited by 23
  14. 2023Breaking the Trilemma of Privacy, Utility, Efficiency via Controllable Machine Unlearningcited by 21
  15. 2024Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clusteringcited by 21
  16. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  17. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reductioncited by 15
  18. 2024Customized Multiple Clustering via Multi-Modal Subspace Proxy Learningcited by 12
  19. 2024NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrativecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 5.

28 papers.

Sort:

By area:

  1. 2025KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systemscited by 36
  2. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  3. 2025Pretrained Image-Text Models are Secretly Video Captionerscited by 11
  4. 2025CEC-Zero: Zero-Supervision Character Error Correction with Self-Generated Rewardscited by 10
  5. 2026Does Memory Need Graphs? A Unified Framework and Empirical Analysis for Long-Term Dialog Memorycited by 9
  6. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  7. 2025MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decodingcited by 7
  8. 2025FakeSV-VLM: Taming VLM for Detecting Fake Short-Video News via Progressive Mixture-Of-Experts Adaptercited by 7
  9. 2025OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learningcited by 7
  10. 2025fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understandingcited by 7
  11. 2025Adversarial Cooperative Rationalization: The Risk of Spurious Correlations in Even Clean Datasetscited by 6
  12. 2026Seeing through the Conflict: Transparent Knowledge Conflict Handling in Retrieval-Augmented Generationcited by 6
16 more
  1. 2025A Review of Incorporating Psychological Theories in LLMscited by 5
  2. 2025Music Audio-Visual Question Answering Requires Specialized Multimodal Designscited by 4
  3. 2025Learning Sparsity for Effective and Efficient Music Performance Question Answeringcited by 4
  4. 2025CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoningcited by 4
  5. 2025DongbaMIE: A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictogramscited by 3
  6. 2025GDLLM: A Global Distance-aware Modeling Approach Based on Large Language Models for Event Temporal Relation Extractioncited by 2
  7. 2025Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusioncited by 1
  8. 2025MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusioncited by 1
  9. 2026Q-Hawkeye: Reliable Visual Policy Optimization for Image Quality Assessmentcited by 1
  10. 2025Multi-Modal Machine Learning Framework for Predicting Early Recurrence of Brain Tumors Using MRI and Clinical Biomarkerscited by 0
  11. 2025A Multimodal Deep Learning Framework for Early Diagnosis of Liver Cancer via Optimized BiLSTM-AM-VMD Architecturecited by 0
  12. 2025ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answeringcited by 0
  13. 2025CPCLDETECTOR: Knowledge Enhancement and Alignment Selection for Chinese Patronizing and Condescending Language Detectioncited by 0
  14. 2025Context-Aware Pseudo-Label Scoring for Zero-Shot Video Summarizationcited by 0
  15. 2026UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memorycited by 0
  16. 2026LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perceptioncited by 0