# Modern Hopfield Networks and Attention for Immune Repertoire Classification

arXiv: https://arxiv.org/abs/2007.13505 · HTML: https://mlsanity.com/paper/2007.13505/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2020.

## Built on by (most cited first, 30 of 53)

- [CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP](https://mlsanity.com/paper/2110.11316/) (2021)
- [Associative Memories via Predictive Coding](https://mlsanity.com/paper/2109.08063/) (2021)
- [Conformal Prediction for Time Series with Modern Hopfield Networks](https://mlsanity.com/paper/2303.12783/) (2023)
- [On Sparse Modern Hopfield Model](https://mlsanity.com/paper/2309.12673/) (2023)
- [History Compression via Language Models in Reinforcement Learning](https://mlsanity.com/paper/2205.12258/) (2022)
- [Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models](https://mlsanity.com/paper/2404.03827/) (2024)
- [Outlier-Efficient Hopfield Layers for Large Transformer-Based Models](https://mlsanity.com/paper/2404.03828/) (2024)
- [STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction](https://mlsanity.com/paper/2312.17346/) (2023)
- [Hierarchical Associative Memory](https://mlsanity.com/paper/2107.06446/) (2021)
- [Txt2Img-MHN: Remote Sensing Image Generation from Text Using Modern Hopfield Networks](https://mlsanity.com/paper/2208.04441/) (2022)
- [Learning to Modulate pre-trained Models in RL](https://mlsanity.com/paper/2306.14884/) (2023)
- [Context-enriched molecule representations improve few-shot drug discovery](https://mlsanity.com/paper/2305.09481/) (2023)
- [BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model](https://mlsanity.com/paper/2404.03830/) (2024)
- [Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Action](https://mlsanity.com/paper/2405.17822/) (2024)
- [Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes](https://mlsanity.com/paper/2410.23126/) (2024)
- [Hopular: Modern Hopfield Networks for Tabular Data](https://mlsanity.com/paper/2206.00664/) (2022)
- [Simplicial Hopfield networks](https://mlsanity.com/paper/2305.05179/) (2023)
- [Nonparametric Modern Hopfield Models](https://mlsanity.com/paper/2404.03900/) (2024)
- [BayesPCN: A Continually Learnable Predictive Coding Associative Memory](https://mlsanity.com/paper/2205.09930/) (2022)
- [Trusted Artificial Intelligence: Towards Certification of Machine Learning Applications](https://mlsanity.com/paper/2103.16910/) (2021)
- [Modern Hopfield Networks for Few- and Zero-Shot Reaction Template Prediction](https://mlsanity.com/paper/2104.03279/) (2021)
- [Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models](https://mlsanity.com/paper/2309.16750/) (2023)
- [Reproducibility in Multiple Instance Learning: A Case For Algorithmic Unit Tests](https://mlsanity.com/paper/2310.17867/) (2023)
- [Energy-based Hopfield Boosting for Out-of-Distribution Detection](https://mlsanity.com/paper/2405.08766/) (2024)
- [Modern Hopfield Networks Require Chain-of-Thought to Solve mathsfNC1-Hard Problems](https://mlsanity.com/paper/2412.05562/) (2024)
- [Long-Term Visual Object Tracking with Event Cameras: An Associative Memory Augmented Tracker and A Benchmark Dataset](https://mlsanity.com/paper/2403.05839/) (2024)
- [Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inference](https://mlsanity.com/paper/2306.12465/) (2023)
- [Sequence Learning Using Equilibrium Propagation](https://mlsanity.com/paper/2209.09626/) (2022)
- [Recent advances in artificial intelligence for retrosynthesis](https://mlsanity.com/paper/2301.05864/) (2023)
- [Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking](https://mlsanity.com/paper/2406.12220/) (2024)

## Abstract (arXiv)

A central mechanism in machine learning is to identify, store, and recognize patterns. How to learn, access, and retrieve such patterns is crucial in Hopfield networks and the more recent transformer architectures. We show that the attention mechanism of transformer architectures is actually the update rule of modern Hopfield networks that can store exponentially many patterns. We exploit this high storage capacity of modern Hopfield networks to solve a challenging multiple instance learning (MIL) problem in computational biology: immune repertoire classification. Accurate and interpretable machine learning methods solving this problem could pave the way towards new vaccines and therapies, which is currently a very relevant research topic intensified by the COVID-19 crisis. Immune repertoire classification based on the vast number of immunosequences of an individual is a MIL problem with an unprecedentedly massive number of instances, two orders of magnitude larger than currently considered problems, and with an extremely low witness rate. In this work, we present our novel method DeepRC that integrates transformer-like attention, or equivalently modern Hopfield networks, into deep learning architectures for massive MIL such as immune repertoire classification. We demonstrate that DeepRC outperforms all other methods with respect to predictive performance on large-scale experiments, including simulated and real-world virus infection data, and enables the extraction of sequence motifs that are connected to a given disease class. Source code and datasets: https://github.com/ml-jku/DeepRC