# Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding

arXiv: https://arxiv.org/abs/2502.06020 · HTML: https://mlsanity.com/paper/2502.06020/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 9 Feb 2025.

## Builds on (most cited first, 30 of 31)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding](https://mlsanity.com/paper/2404.05726/) (2024)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [Large Language Models with Controllable Working Memory](https://mlsanity.com/paper/2211.05110/) (2022)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [Long Movie Clip Classification with State-Space Video Models](https://mlsanity.com/paper/2204.01692/) (2022)
- [Selective Structured State-Spaces for Long-Form Video Understanding](https://mlsanity.com/paper/2303.14526/) (2023)
- [Condensed Movies: Story Based Retrieval with Contextual Embeddings](https://mlsanity.com/paper/2005.04208/) (2020)
- [Timeception for Complex Action Recognition](https://mlsanity.com/paper/1812.01289/) (2018)
- [VindLU: A Recipe for Effective Video-and-Language Pretraining](https://mlsanity.com/paper/2212.05051/) (2022)
- [Vision Transformers are Parameter-Efficient Audio-Visual Learners](https://mlsanity.com/paper/2212.07983/) (2022)
- [Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench](https://mlsanity.com/paper/2410.22108/) (2024)
- [TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding](https://mlsanity.com/paper/2310.19060/) (2023)
- [InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning](https://mlsanity.com/paper/2409.12568/) (2024)
- [Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks](https://mlsanity.com/paper/2311.05152/) (2023)
- [Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering](https://mlsanity.com/paper/2310.06238/) (2023)
- [Bootstrapping Vision-Language Learning with Decoupled Language Pre-training](https://mlsanity.com/paper/2307.07063/) (2023)
- [Working Memory Capacity of ChatGPT: An Empirical Study](https://mlsanity.com/paper/2305.03731/) (2023)
- [Breaking the Trilemma of Privacy, Utility, Efficiency via Controllable Machine Unlearning](https://mlsanity.com/paper/2310.18574/) (2023)
- [Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering](https://mlsanity.com/paper/2404.15655/) (2024)
- [Learning Video Context as Interleaved Multimodal Sequences](https://mlsanity.com/paper/2407.21757/) (2024)
- [Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction](https://mlsanity.com/paper/2310.03291/) (2023)
- [Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning](https://mlsanity.com/paper/2411.03978/) (2024)

## Built on by (most cited first, 28 of 28)

- [KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systems](https://mlsanity.com/paper/2502.07350/) (2025)
- [3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model](https://mlsanity.com/paper/2505.22657/) (2025)
- [Pretrained Image-Text Models are Secretly Video Captioners](https://mlsanity.com/paper/2502.13363/) (2025)
- [CEC-Zero: Zero-Supervision Character Error Correction with Self-Generated Rewards](https://mlsanity.com/paper/2512.23971/) (2025)
- [Does Memory Need Graphs? A Unified Framework and Empirical Analysis for Long-Term Dialog Memory](https://mlsanity.com/paper/2601.01280/) (2026)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding](https://mlsanity.com/paper/2505.15946/) (2025)
- [FakeSV-VLM: Taming VLM for Detecting Fake Short-Video News via Progressive Mixture-Of-Experts Adapter](https://mlsanity.com/paper/2508.19639/) (2025)
- [OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning](https://mlsanity.com/paper/2509.01644/) (2025)
- [fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding](https://mlsanity.com/paper/2511.21760/) (2025)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Adversarial Cooperative Rationalization: The Risk of Spurious Correlations in Even Clean Datasets](https://mlsanity.com/paper/2505.02118/) (2025)
- [Seeing through the Conflict: Transparent Knowledge Conflict Handling in Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.06842/) (2026)
- [Music Audio-Visual Question Answering Requires Specialized Multimodal Designs](https://mlsanity.com/paper/2505.20638/) (2025)
- [Learning Sparsity for Effective and Efficient Music Performance Question Answering](https://mlsanity.com/paper/2506.01319/) (2025)
- [CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoning](https://mlsanity.com/paper/2512.19554/) (2025)
- [DongbaMIE: A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictograms](https://mlsanity.com/paper/2503.03644/) (2025)
- [GDLLM: A Global Distance-aware Modeling Approach Based on Large Language Models for Event Temporal Relation Extraction](https://mlsanity.com/paper/2508.20828/) (2025)
- [Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion](https://mlsanity.com/paper/2504.03135/) (2025)
- [MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusion](https://mlsanity.com/paper/2509.17446/) (2025)
- [Q-Hawkeye: Reliable Visual Policy Optimization for Image Quality Assessment](https://mlsanity.com/paper/2601.22920/) (2026)
- [Multi-Modal Machine Learning Framework for Predicting Early Recurrence of Brain Tumors Using MRI and Clinical Biomarkers](https://mlsanity.com/paper/2509.01161/) (2025)
- [A Multimodal Deep Learning Framework for Early Diagnosis of Liver Cancer via Optimized BiLSTM-AM-VMD Architecture](https://mlsanity.com/paper/2509.01164/) (2025)
- [ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answering](https://mlsanity.com/paper/2509.16680/) (2025)
- [CPCLDETECTOR: Knowledge Enhancement and Alignment Selection for Chinese Patronizing and Condescending Language Detection](https://mlsanity.com/paper/2509.18562/) (2025)
- [Context-Aware Pseudo-Label Scoring for Zero-Shot Video Summarization](https://mlsanity.com/paper/2510.17501/) (2025)
- [UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory](https://mlsanity.com/paper/2608.30508/) (2026)
- [LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception](https://mlsanity.com/paper/2609.39938/) (2026)

## Abstract (arXiv)

Multimodal foundation models (MFMs) have demonstrated significant success in tasks such as visual captioning, question answering, and image-text retrieval. However, these models face inherent limitations due to their finite internal capacity, which restricts their ability to process extended temporal sequences, a crucial requirement for comprehensive video and audio analysis. To overcome these challenges, we introduce a specialized cognitive module, temporal working memory (TWM), which aims to enhance the temporal modeling capabilities of MFMs. It selectively retains task-relevant information across temporal dimensions, ensuring that critical details are preserved throughout the processing of video and audio content. The TWM uses a query-guided attention approach to focus on the most informative multimodal segments within temporal sequences. By retaining only the most relevant content, TWM optimizes the use of the model's limited capacity, enhancing its temporal modeling ability. This plug-and-play module can be easily integrated into existing MFMs. With our TWM, nine state-of-the-art models exhibit significant performance improvements across tasks such as video captioning, question answering, and video-text retrieval. By enhancing temporal modeling, TWM extends the capability of MFMs to handle complex, time-sensitive data effectively. Our code is available at https://github.com/xid32/NAACL_2025_TWM.