mlsanityData to 14 Oct 2026

The Concept Allocation Zone: Tracking How Concepts Form Across Transformer Depth

arXiv 2605.24856 ↗First posted 24 May 2026cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Concept formation in transformer language models is a depth-extended process, not a single-layer event: a concept becomes separable across one or more contiguous regions of the residual stream - its Concept Allocation Zone (CAZ). A CAZ is not a concept but the depth segment where the model organizes its geometry to make one separable - concepts may share a CAZ, and typically span multiple across depth; the companion GEM paper shows the separating direction continues to rotate within a CAZ before stabilizing past its boundary. We formalize the CAZ through three layer-wise metrics - Separation, Concept Coherence, and Concept Velocity - with automated boundary detection that applies no significance threshold to CAZ membership (every segment is a CAZ; "strong" vs. "gentle" is score, never a binary cut). Empirical validation across 35 models, 8 architectural families, and 7 concepts shows the separation curve S(l) is frequently multimodal, and scored detection surfaces a further category of subtle allocation regions ("gentle CAZes") invisible to standard peak detection. The framework generates seven testable predictions; its contribution is the instrument and the phenomena it surfaces - the scored detector, the three metrics, and the multimodal/gentle-CAZ findings - not the predictions themselves. Released as the open-source rosetta_tools library (v1.3.1).

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). First posted on arXiv 24 May 2026.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.