# Memory Encoding Model

arXiv: https://arxiv.org/abs/2308.01175 · HTML: https://mlsanity.com/paper/2308.01175/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 2 Aug 2023.

## Builds on (most cited first, 11 of 11)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [BOLD5000: A public fMRI dataset of 5000 images](https://mlsanity.com/paper/1809.01281/) (2018)
- [Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision](https://mlsanity.com/paper/1608.03425/) (2016)
- [The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes](https://mlsanity.com/paper/2301.03198/) (2023)
- [The Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion](https://mlsanity.com/paper/2104.13714/) (2021)
- [The Sensorium competition on predicting large-scale mouse primary visual cortex activity](https://mlsanity.com/paper/2206.08666/) (2022)
- [Retinotopy Inspired Brain Encoding Model and the All-for-One Training Recipe](https://mlsanity.com/paper/2307.14021/) (2023)

## Built on by (most cited first, 7 of 7)

- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [MindSimulator: Exploring Brain Concept Localization via Synthetic FMRI](https://mlsanity.com/paper/2503.02351/) (2025)
- [A foundation model of vision, audition, and language for in-silico neuroscience](https://mlsanity.com/paper/2605.04326/) (2026)
- [Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion](https://mlsanity.com/paper/2411.09723/) (2024)
- [SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning](https://mlsanity.com/paper/2508.10298/) (2025)
- [Predicting Brain Responses To Natural Movies With Multimodal LLMs](https://mlsanity.com/paper/2507.19956/) (2025)
- [A Parameter-efficient Multi-subject Model for Predicting fMRI Activity](https://mlsanity.com/paper/2308.02351/) (2023)

## Abstract (arXiv)

We explore a new class of brain encoding model by adding memory-related information as input. Memory is an essential brain mechanism that works alongside visual stimuli. During a vision-memory cognitive task, we found the non-visual brain is largely predictable using previously seen images. Our Memory Encoding Model (Mem) won the Algonauts 2023 visual brain competition even without model ensemble (single model score 66.8, ensemble score 70.8). Our ensemble model without memory input (61.4) can also stand a 3rd place. Furthermore, we observe periodic delayed brain response correlated to 6th-7th prior image, and hippocampus also showed correlated activity timed with this periodicity. We conjuncture that the periodic replay could be related to memory mechanism to enhance the working memory.