# Predicted Cortex Is Not a Domain-General Prior: A Matched-Control Audit of Brain-Encoding Features for Video Memorability

arXiv: https://arxiv.org/abs/2607.16292 · HTML: https://mlsanity.com/paper/2607.16292/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 10 references that are in the map.
- First posted on arXiv 12 Jul 2026.

## Builds on (most cited first, 4 of 4)

- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [VideoMem: Constructing, Analyzing, Predicting Short-term and Long-term Video Memorability](https://mlsanity.com/paper/1812.01973/) (2018)
- [Multimodal Memorability: Modeling Effects of Semantics and Decay on Video Memorability](https://mlsanity.com/paper/2009.02568/) (2020)
- [Insights from the Algonauts 2025 Winners](https://mlsanity.com/paper/2508.10784/) (2025)

## Abstract (arXiv)

Brain-encoding foundation models predict fMRI responses to video, audio and text well enough to win the Algonauts 2025 challenge. We ask whether their predicted responses, obtained with no scanner, are a useful feature lens for a human-behavior task: forecasting short-video memorability. Each clip is projected into TRIBE v2's predicted cortical space and scored by ridge regression against a matched control, the model's own V-JEPA2 visual backbone taken before the brain projection. The answer is dataset-dependent. Within Memento10k (499 clips) the backbone wins (Spearman 0.594 vs 0.544); within VideoMem (820 clips) the brain projection wins (0.415 vs 0.368). Because the claim is that the ordering reverses, we test the reversal itself: the dataset-by-representation interaction is +0.097, 95% CI [+0.032, +0.160], two-sided bootstrap p=0.001, and over 10 cross-validation seeds the datasets separate completely (0/10 seeds favor the brain projection on Memento10k, 10/10 on VideoMem). Cross-dataset transfer inherits the split: Memento10k->VideoMem the brain projection wins (+0.076); the reverse loses heavily (-0.311). The VideoMem advantage is not a sample-size artifact (it survives matched training size and PCA-then-ridge) and not mere compression (a compressed, heavily regularized or transfer-tuned backbone stays below it). Predicted-brain features thus carry a small but real memorability signal the backbone misses on one dataset and not the other: a dataset-specific representation, not a domain-general prior. A vision-orthogonal component (partial Spearman 0.19, permutation p=2.5e-4) localizes to ventral occipito-temporal cortex, and predicted BOLD dynamics add nothing beyond the time-average because 3-4 samples per clip cannot resolve the sub-second late memorability response. Our pre-specified within-dataset hypothesis returned NO-GO; the reversal is what survived.