mlsanityData to 14 Oct 2026

Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective

arXiv 2605.27476 ↗First posted 26 May 2026cs.LG cs.AIcited by 0

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

We characterize the pre-softmax attention matrix QK^top in transformers as an associative memory matrix encoding pairwise associations between input features. By decomposing this matrix into its symmetric and skew-symmetric parts, we interpret the symmetric component as governing the structure of the energy landscape, and the skew-symmetric component as driving circulation on that landscape. Leveraging the energy formulation induced by the symmetric component, we derive Hopfield-style stability measures that quantify the stability of retrieved features. We observe meaningful correlations between Hopfield-style stability measures and the fidelity-diversity trade-offs in generation. Finally, we propose a controllable knob to modulate this trade-off by modifying the circulation of the underlying dynamics. Code is available at our GitHub (https://github.com/hyeon-cho/Attention-Symmetric-Decomposition).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 32 of its 49 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

32 of 49 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Memorization to Generalization: Emergence of Diffusion Models from Associative Memory27 May
  2. 2025PLADIS: Pushing the Limits of Attention in Diffusion Models at Inference Time by Leveraging Sparsity10 Mar
  3. 2024EDT: An Efficient Diffusion Transformer Framework Inspired by Human-like Sketching31 Oct
  4. 2024Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention1 Aug
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  6. 2023In search of dispersed memories: Generative diffusion models are associative memory networks29 Sep
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  8. 2022Scalable Diffusion Models with Transformers19 Dec
  9. 2022Classifier-Free Diffusion Guidance26 Jul
  10. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  11. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
1 more
  1. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Flow Matching for Generative Modelingcited by 5,702
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  10. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  11. 2018Self-Attention Generative Adversarial Networkscited by 809
  12. 2020Hopfield Networks is All You Needcited by 475
7 more
  1. 2018Diverse Image-to-Image Translation via Disentangled Representationscited by 296
  2. 2016Dense Associative Memory for Pattern Recognitioncited by 236
  3. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  4. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  5. 2025Dense Associative Memory with Epanechnikov Energycited by 11
  6. 2024Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!cited by 9
  7. 2024Self-attention as an attractor network: transient memories without backpropagationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Geometry-Aware Attention Guidance for Diffusion Models via Modern Hopfield Dynamics
  2. 2026The Diffusion-Attention Connection
  3. 2024Dynamical Mean-Field Theory of Self-Attention Neural Networks
  4. 2026Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy
  5. 2026Attention Sinks in Diffusion Transformers: A Causal Analysis
  6. 2026Hierarchical Prototype Emergence in Modern Hopfield Models
  7. 2026Quantifying LLM Attention-Head Stability: Implications for Circuit Universality
  8. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Models