Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective

We characterize the pre-softmax attention matrix $\mathbf{Q}\mathbf{K}^\top$ in transformers as an associative memory matrix encoding pairwise associations between input features. By decomposing this matrix into its symmetric and skew-symmetric parts, we interpret the symmetric component as governing the structure of the energy landscape, and the skew-symmetric component as driving circulation on that landscape.

Leveraging the energy formulation induced by the symmetric component, we derive Hopfield-style stability measures that quantify the stability of retrieved features. We observe meaningful correlations between these measures and the fidelity–diversity trade-off in generation. Finally, we propose a controllable knob to modulate this trade-off by modifying the circulation of the underlying dynamics.

The metastable-mixture problem

Diffusion models often blend incompatible features—materials mixed across distinct objects, anatomically implausible structures, or attribute leakage. The same global connectivity in attention that helps compositional generation can also settle into incoherent combinations of distinct patterns.

Most prior analyses operate at a token-wise level, treating attention as retrieval and reading off a single token at a time. That view misses the interaction dynamics encoded by the attention matrix itself: how features collectively settle, and whether they settle into fixed points or cycles.

We take an associative-memory view of $\mathbf{Q}\mathbf{K}^\top$, building on the observation that transformer self-attention approximates the update rule of a modern Hopfield network. From this lens, spurious mixing is entrapment in metastable states—local energy minima where the model rests on an incoherent superposition of patterns.

Associative memory view of self-attention

Treat the input features $\mathbf{X}\in\mathbb{R}^{L\times d_{\rm in}}$ as a collection of $d_{\rm in}$ feature vectors $\boldsymbol{x}^{(i)}\in\mathbb{R}^L$. The pre-softmax attention $\mathbf{Q}\mathbf{K}^\top = \mathbf{X}\mathbf{W}\mathbf{X}^\top$ then reads as a pairwise association strength between feature pairs $(i,j)$—the natural object in a Hopfield-style associative memory.

Symmetric / skew decomposition

Every pre-softmax attention matrix uniquely decomposes:

Symmetric component $\mathbf{S}$

Defines a Hopfield-style energy $E_{\mathbf{X}}(\xi)$ on retrieved features. Its local minima are stable attractors, including some incoherent metastable mixtures.

Skew component $\mathbf{N}$

Satisfies $\mathbf{u}^\top\mathbf{N}\mathbf{u}=0$, so it contributes nothing to the energy. Instead, it supplies a directional circulation field that perturbs the stable structure.

Hopfield stability measures

From the symmetric energy $E_{\mathbf{X}}(\xi)$ and the induced local field $\boldsymbol{h}_{\mathbf{X}}(\xi)$, we obtain three complementary internal diagnostics:

• Hopfield energy $E_{\mathbf{X}}$ measures global self-consistency. Lower energy indicates a more coherent retrieval.
• Instability fraction $r_{\mathbf{X}}$ measures the share of features in local conflict.
• Alignment score $\mathbf{Align}_{\mathbf{X}}$ measures directional agreement between the retrieval and its driving field.

These measures are computed purely from the model’s own attention and distinguish coherent retrievals from metastable mixtures.

Spearman rank correlation $\rho$ between the three internal stability measures and standard external metrics, measured across each block of the SDXL UNet over 1K MSCOCO prompts, reveals a consistent pattern.

• Aesthetic Score positively correlates with stability at every depth.
• LPIPS Diversity inversely correlates with stability: metastable mixtures are a source of both diversity and hallucination.
• CLIPScore and ImageReward show depth-dependent behavior, with the strongest signals in the Down and Up blocks.

On unstable samples, skew perturbation breaks spurious mixtures and recovers cleaner, object-centric structure. On stable samples, it introduces local variation in texture, background, and composition—revealing the operating-point trade-off.