# Future Video Generation Better Aligns with the Human Visual Cortex than Observed Video

arXiv: https://arxiv.org/abs/2609.38819 · HTML: https://mlsanity.com/paper/2609.38819/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 22 of its 57 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 22 of 22)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [Your Diffusion Model is Secretly a Zero-Shot Classifier](https://mlsanity.com/paper/2303.16203/) (2023)
- [Denoising Diffusion Autoencoders are Unified Self-supervised Learners](https://mlsanity.com/paper/2303.09769/) (2023)
- [Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance](https://mlsanity.com/paper/2403.17377/) (2024)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Helios: Real Real-Time Long Video Generation Model](https://mlsanity.com/paper/2603.04379/) (2026)
- [Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later](https://mlsanity.com/paper/2303.02490/) (2023)
- [Cognitive computational neuroscience](https://mlsanity.com/paper/1807.11819/) (2018)
- [Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling](https://mlsanity.com/paper/2411.18664/) (2024)
- [Emergent Temporal Correspondences from Video Diffusion Transformers](https://mlsanity.com/paper/2506.17220/) (2025)
- [Video-Mirai: Autoregressive Video Diffusion Models Need Foresight](https://mlsanity.com/paper/2606.03971/) (2026)
- [The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show](https://mlsanity.com/paper/2606.05328/) (2026)
- [Denoise to Track: Harnessing Video Diffusion Priors for Robust Correspondence](https://mlsanity.com/paper/2512.04619/) (2025)
- [Only Brains Align with Brains: Cross-Region Alignment Patterns Expose Limits of Normative Models](https://mlsanity.com/paper/2604.21780/) (2026)
- [Diffusion-Based Action Recognition Generalizes to Untrained Domains](https://mlsanity.com/paper/2509.08908/) (2025)
- [Gen4U: Unifying Video Generation and Understanding via Diffusion](https://mlsanity.com/paper/2607.06856/) (2026)

## Abstract (arXiv)

Studying the alignment between the internal representations of vision models and the responses of the visual cortex to the same observed visual stimuli has enabled us to better understand human visual processing. However, studies so far have largely overlooked the fact that the human brain not only processes observed visual stimuli, but also predicts upcoming stimuli based on what has been observed. Accordingly, we hypothesize that internal representations for generating future video frames are better aligned with the predictive nature of human visual processing than representations of the observed video itself. To this end, we compare the alignment between human video-watching fMRI responses in the visual cortex and the internal representations from two types of video diffusion models, an autoregressive (AR) model and its non-AR base model. We first conduct a within-model analysis of the AR video diffusion model and show that the representations for future video generation align better with the visual cortex than the representations of the observed video. We then compare the internal representations of the AR model with those of its non-AR base model and again show that the representations for future video generation align better with the visual cortex than the representations for observed video reconstruction by the base model. Specifically, the alignment of observed video reconstruction is concentrated in lower-order visual cortex, whereas that of future video generation is concentrated in higher-order visual cortex. Finally, we show in a human behavioral experiment that humans prefer videos generated by amplifying the contributions of individual layers that align better with the visual cortex.