Uncovering hidden geometry in Transformers via disentangling position and context

Transformers are widely used to extract semantic meanings from input tokens, yet they usually operate as black-box models. In this paper, we present a simple yet informative decomposition of hidden states (or embeddings) of trained transformers into interpretable components. For any layer, embedding vectors of input sequence samples are represented by a tensor boldsymbolh ∈ ℝC × T × d. Given embedding vector boldsymbolhc,t ∈ ℝd at sequence position t ≤ T in a sequence (or context) c ≤ C, extracting the mean effects yields the decomposition [ boldsymbolhc,t = boldsymbolμ + post + ctxc + residc,t ] where boldsymbolμ is the global mean vector, post and ctxc are the mean vectors across contexts and across positions respectively, and residc,t is the residual vector. For popular transformer architectures and diverse text datasets, empirically we find pervasive mathematical structure: (1) (post)t forms a low-dimensional, continuous, and often spiral shape across layers, (2) (ctxc)c shows clear cluster structure that falls into context topics, and (3) (post)t and (ctxc)c are mutually nearly orthogonal. We argue that smoothness is pervasive and beneficial to transformers trained on languages, and our decomposition leads to improved model interpretability.