# Hyperparameter Transfer for Dense Associative Memories

arXiv: https://arxiv.org/abs/2605.10164 · HTML: https://mlsanity.com/paper/2605.10164/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 16 references that are in the map.
- First posted on arXiv 11 May 2026.

## Builds on (most cited first, 7 of 7)

- [Feature Learning in Infinite-Width Neural Networks](https://mlsanity.com/paper/2011.14522/) (2020)
- [A Spectral Condition for Feature Learning](https://mlsanity.com/paper/2310.17813/) (2023)
- [Scaling Exponents Across Parameterizations and Optimizers](https://mlsanity.com/paper/2407.05872/) (2024)
- [Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer](https://mlsanity.com/paper/2502.02531/) (2025)
- [Modern Methods in Associative Memory](https://mlsanity.com/paper/2507.06211/) (2025)
- [Hyperparameter Transfer with Mixture-of-Expert Layers](https://mlsanity.com/paper/2601.20205/) (2026)
- [NRGPT: An Energy-based Alternative for GPT](https://mlsanity.com/paper/2512.16762/) (2025)

## Abstract (arXiv)

Dense Associative Memory (DenseAM) is a promising family of AI architectures that is represented by a neural network performing temporal dynamics on an energy landscape. While hyperparameter transfer methods are well-studied for feed-forward networks, these methods have not been developed for settings in which weights are shared across layers and within the layer, which is common in DenseAMs. Additionally, DenseAMs utilize rapidly peaking activation functions that are rarely used in feed-forward architectures. The confluence of these aspects makes DenseAM a challenging framework for using existing methods for hyperparameter transfer. Our work initiates the development of hyperparameter transfer methods for this class of models. We derive explicit prescriptions for how the hyperparameters tuned on small models can be transferred to models trained at scale. We demonstrate excellent agreement between these theoretical findings and empirical results.