How does longer temporal context enhance multimodal narrative video processing in the brain?
Not on the map. Why?
Abstract (arXiv)
Understanding how humans and artificial intelligence systems process complex narrative videos is a fundamental challenge at the intersection of neuroscience and machine learning. This study investigates how the temporal context length of video clips (3--24 s clips) and the narrative-task prompting shape brain-model alignment during naturalistic movie watching. Using fMRI recordings from participants viewing full-length movies, we examine how brain regions sensitive to narrative context dynamically represent information over varying timescales and how these neural patterns align with model-derived features. We find that increasing clip duration substantially improves brain alignment for multimodal large language models (MLLMs), whereas unimodal video models show little to no gain. Further, shorter temporal windows align with perceptual and early language regions, while longer windows preferentially align higher-order integrative regions, mirrored by a layer-to-cortex hierarchy in MLLMs. Finally, experiments with four narrative-task prompts show that they elicit task-specific, region-dependent brain alignment patterns and context-dependent shifts in clip-level tuning in higher-order regions. Our work positions long-form narrative movies as a principled testbed for studying long-timescale temporal integration in long-context MLLMs and its relationship to cortical responses during narrative comprehension.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 59 references that are in the map. First posted on arXiv 7 Feb 2026.
Builds on
17 of 59 references.
References · 17
- 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
- 2025Qwen2.5-Omni Technical Reportcited by 704
- 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
- 2018Machine Theory of Mindcited by 222
- 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
- 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
- 2022Self-supervised models of audio effectively explain human cortical responses to speechcited by 37
- 2023Brain encoding models based on multimodal transformers can transfer across language and visioncited by 31
- 2022Training language models to summarize narratives improves brain alignmentcited by 26
- 2023Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)cited by 20
- 2023Speech language models lack important brain-relevant semanticscited by 19
5 more
- 2024Revealing Vision-Language Integration in the Brain with Multimodal Networkscited by 11
- 2023Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Braincited by 9
- 2025DATE: Dynamic Absolute Time Enhancement for Long Video Understandingcited by 9
- 2025Multi-modal brain encoding models for multi-modal stimulicited by 8
- 2025Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)cited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Task-conditioned probing of instruction-tuned multimodal LLMs: Region-specific brain alignment patterns under naturalistic stimuli
- 2022Visual representations in the human brain are aligned with large language models
- 2025Scaling and context steer LLMs along the same computational path as the human brain
- 2026Brain alignment of reasoning and action representations from vision-language and action models during naturalistic gameplay
- 2026The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language models
- 2023Attention, not scale, drives human-AI alignment in multimodal language prediction
- 2025Do Large Language Models Think Like the Brain? Sentence-Level Evidences from Layer-Wise Embeddings and fMRI
- 2025Beyond Single Frames: Can LMMs Comprehend Temporal and Contextual Narratives in Image Sequences?