mlsanityData to 7 Oct 2026

Energy Transformer

arXiv 2302.07253 ↗First posted 14 Feb 2023cs.LG cond-mat.dis-nn cs.CVcited by 69

Not on the map. Why?

Abstract (arXiv)

Our work combines aspects of three promising paradigms in machine learning, namely, attention mechanism, energy-based models, and associative memory. Attention is the power-house driving modern deep learning successes, but it lacks clear theoretical foundations. Energy-based models allow a principled approach to discriminative and generative tasks, but the design of the energy functional is not straightforward. At the same time, Dense Associative Memory models or Modern Hopfield Networks have a well-established theoretical foundation, and allow an intuitive design of the energy function. We propose a novel architecture, called the Energy Transformer (or ET for short), that uses a sequence of attention layers that are purposely designed to minimize a specifically engineered energy function, which is responsible for representing the relationships between the tokens. In this work, we introduce the theoretical foundations of ET, explore its empirical capabilities using the image completion task, and obtain strong quantitative results on the graph anomaly detection and graph classification tasks.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 40 of its 71 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

40 of 71 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2017Graph Attention Networkscited by 8,854
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2017Decoupled Weight Decay Regularizationcited by 1,722
  10. 2019Simplifying Graph Convolutional Networkscited by 1,470
  11. 2019When Does Label Smoothing Help?cited by 985
  12. 2020TUDataset: A collection of benchmark datasets for learning with graphscited by 718
28 more
  1. 2020A Generalization of Transformer Networks to Graphscited by 653
  2. 2019Deep Equilibrium Modelscited by 603
  3. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  4. 2020Hopfield Networks is All You Needcited by 461
  5. 2021A Survey of Transformerscited by 399
  6. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  7. 2020Enhancing Graph Neural Network-based Fraud Detectors against Camouflaged Fraudsterscited by 248
  8. 2016Dense Associative Memory for Pattern Recognitioncited by 230
  9. 2022Rethinking Graph Neural Networks for Anomaly Detectioncited by 185
  10. 2019The Evolved Transformercited by 180
  11. 2019ASAP: Adaptive Structure Aware Pooling for Learning Hierarchical Graph Representationscited by 156
  12. 2021GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graphcited by 143
  13. 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 113
  14. 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 110
  15. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  16. 2013From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviewscited by 107
  17. 2020Gradient Centralization: A New Optimization Technique for Deep Neural Networkscited by 82
  18. 2020Alleviating the Inconsistency Problem of Applying Graph Neural Network to Fraud Detectioncited by 81
  19. 2019Learning metrics for persistence-based summaries and applications for graph classificationcited by 57
  20. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  21. 2019Universal Graph Transformer Self-Attention Networkscited by 41
  22. 2021Hierarchical Associative Memorycited by 34
  23. 2019Hierarchical Representation Learning in Graph Neural Networks with Node Decimation Poolingcited by 30
  24. 2022Transformers from an Optimization Perspectivecited by 27
  25. 2020Bridging the Gap Between Spectral and Spatial Domains in Graph Neural Networkscited by 26
  26. 2021A New Perspective on the Effects of Spectrum in Graph Neural Networkscited by 24
  27. 2021A remark on a paper of Krotov and Hopfield [arXiv:2008.06996]cited by 11
  28. 2019From Static to Dynamic Anomaly Detection with Application to Power System Cyber Securitycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 17; 2025: 20; 2026: 23.

69 papers.

Sort:

By area:

  1. 2023Large Language Modelscited by 774
  2. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  3. 2023White-Box Transformers via Sparse Rate Reductioncited by 77
  4. 2023On Sparse Modern Hopfield Modelcited by 41
  5. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  6. 2024Artificial Kuramoto Oscillatory Neuronscited by 37
  7. 2023STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Predictioncited by 35
  8. 2023Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Modelscited by 30
  9. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
  10. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
  11. 2024Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codescited by 19
  12. 2024Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Modelscited by 18
38 more
  1. 2024Memory Mosaicscited by 16
  2. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  3. 2024Beyond Scaling Laws: Understanding Transformer Performance with Associative Memorycited by 12
  4. 2025Operator Learning for Reconstructing Flow Fields from Sparse Measurements: an Energy Transformer Approachcited by 11
  5. 2024Dense Associative Memory Through the Lens of Random Featurescited by 8
  6. 2025In-context denoising with one-layer transformers: connections between attention and associative memory retrievalcited by 7
  7. 2024Hitchhiker's guide on the relation of Energy-Based Models with other generative models, sampling and statistical physics: a comprehensive reviewcited by 5
  8. 2026Variational Flow Maps: Make Some Noise for One-Step Conditional Generationcited by 5
  9. 2023Associative Transformercited by 4
  10. 2024Dense Hopfield Networks in the Teacher-Student Settingcited by 4
  11. 2024Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breakingcited by 4
  12. 2024An In-depth Investigation of Sparse Rate Reduction in Transformer-like Modelscited by 4
  13. 2025Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobianscited by 4
  14. 2025NRGPT: An Energy-based Alternative for GPTcited by 4
  15. 2023iMixer: hierarchical Hopfield network implies an invertible, implicit and iterative MLP-Mixercited by 3
  16. 2024Zero-Shot Generalization of Vision-Based RL Without Data Augmentationcited by 3
  17. 2025Zero-Shot Visual Generalization in Robot Manipulationcited by 3
  18. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  19. 2025Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principlecited by 3
  20. 2026Krause Synchronization Transformerscited by 3
  21. 2025Hyper-SET: Designing Transformers via Hyperspherical Energy Minimizationcited by 2
  22. 2025Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplingscited by 2
  23. 2025Dense associative memory for Gaussian distributionscited by 2
  24. 2026Deep Clustering with Associative Memoriescited by 2
  25. 2026On the Infinite Width and Depth Limits of Predictive Coding Networkscited by 2
  26. 2026Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energycited by 2
  27. 2026Training-Free Generation of Protein Sequences from Small Family Alignments via Stochastic Attentioncited by 2
  28. 2023Exploring the Temperature-Dependent Phase Transition in Modern Hopfield Networkscited by 1
  29. 2024Cognitively Inspired Energy-Based World Modelscited by 1
  30. 2024Associative memory and dead neuronscited by 1
  31. 2025Neural Learning Rules from Associative Networks Theorycited by 1
  32. 2025The Gaussian-Multinoulli Restricted Boltzmann Machine: A Potts Model Extension of the GRBMcited by 1
  33. 2025A Framework for Non-Linear Attention via Modern Hopfield Networkscited by 1
  34. 2025Wavy Transformercited by 1
  35. 2025On the Role of Hidden States of Modern Hopfield Network in Transformercited by 1
  36. 2026Remapping and navigation of an embedding space via error minimization: a fundamental organizational principle of cognition in natural and artificial systemscited by 1
  37. 2026Hybrid Energy-Based Models for Physical AI: Provably Stable Identification of Port-Hamiltonian Dynamicscited by 1
  38. 2026Energy-Based Dynamical Models for Neurocomputation, Learning, and Optimizationcited by 1

50 most cited of 69 All 69 (JSON)