# Stream separation improves Bregman conditioning in transformers

arXiv: https://arxiv.org/abs/2603.21317 · HTML: https://mlsanity.com/paper/2603.21317/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 10 references that are in the map.
- First posted on arXiv 22 Mar 2026.

## Builds on (most cited first, 7 of 7)

- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [On the Role of Attention Heads in Large Language Model Safety](https://mlsanity.com/paper/2410.13708/) (2024)
- [Superposition Yields Robust Neural Scaling](https://mlsanity.com/paper/2505.10465/) (2025)
- [The Information Geometry of Softmax: Probing and Steering](https://mlsanity.com/paper/2602.15293/) (2026)
- [The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modeling](https://mlsanity.com/paper/2603.07461/) (2026)

## Abstract (arXiv)

Linear methods for steering transformer representations, including probing, activation engineering, and concept erasure, implicitly assume the geometry of representation space is Euclidean. Park et al. [Park et al., 2026] showed that softmax induces a curved Bregman geometry whose metric tensor is the Hessian of the log-normalizer, H(λ) = Cov[γ | λ]. Ignoring this curvature causes Euclidean steering to leak probability mass to unintended tokens. Their analysis applies at the output layer. We measure this Hessian at intermediate layers in a controlled 2x2 design crossing stream separation with per-layer supervision (vocabulary decoding loss at each layer), all at matched vocabulary and parameter count. In standard single-stream transformers, H is severely degenerate at intermediate layers (effective rank 8 in 516 dimensions). Stream separation improves conditioning by up to 22 in effective rank, even without auxiliary supervision. Per-layer supervision helps, but less. The cosine similarity between primal and dual concept directions predicts per-layer steering effectiveness on downstream tasks, with a threshold near 0.3. These results bear on the reliability of linear safety interventions, which depend on the geometry being well-conditioned at the layer where they are applied.