mlsanityData to 14 Oct 2026

Signed-Permutation Coordinate Transport for RMSNorm Transformers

arXiv 2606.31963 ↗First posted 30 Jun 2026cs.LG cs.CL stat.MLcited by 1

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top-k neuron sets, attribution lists, and merge alignments. This is only well posed after fixing the model's residual-stream gauge, which we show is architecture-dependent: LayerNorm residual charts have permutation gauge Sd (up to a global sign flip), while RMSNorm charts with generic per-channel gain have signed-permutation gauge Bd = Sd ltimes ± 1d. Permutation-only alignment is therefore symmetry-incomplete for RMSNorm models. We introduce sign-marginalized Hungarian matching and prove a sharp failure mode: with decorrelated coordinates, raw signed-correlation matching has a structural permutation-accuracy ceiling at the positive-sign fraction of the true gauge, which sign-marginalization removes. We then make coordinate-preserving transport, not function-level merging, the primary object: composing saved-checkpoint local Bd gauges along same-base fine-tuning trajectories recovers 91.1% of cross-run coordinates at 1500 steps versus 60.3% for endpoint matching, and the gain is not explained by merely routing through the base. The recovered gauge transfers tools that permutation-only alignment breaks: TinyLlama SAE reconstruction has NMSE 0.004 under Bd versus 1.08 under Sd; Qwen sentiment steering preserves 95.8% of its effect versus 17.2%; refusal steering reverses sign under Sd; coordinate-preserving merges behave the same way. The same covariance governs stateful training: signed transport of AdamW state preserves the resumed trajectory, while permutation-only state follows a different one from a functionally identical checkpoint. Finally, gauge-sweep audits show index-level interpretability claims are reproducible only relative to an explicit gauge.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 22 of its 28 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

22 of 28 references.

References · 22

  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  3. 2022Editing Models with Task Arithmeticcited by 1,120
  4. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  5. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  6. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  8. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  9. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  10. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  11. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  12. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
10 more
  1. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  2. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  3. 2019Model Fusion via Optimal Transportcited by 219
  4. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  5. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  6. 2025Sparse Autoencoders Trained on the Same Data Learn Different Featurescited by 93
  7. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 86
  8. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 63
  9. 2023Transformer Fusion with Optimal Transportcited by 36
  10. 2025Generalized Linear Mode Connectivity for Transformerscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does Socited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization
  2. 2026Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
  3. 2026Permutation-Equivariant Flow Matching for Alignment-Free Neural Weight Generation
  4. 2025Representational Alignment Across Model Layers and Brain Regions with Multi-Level Optimal Transport
  5. 2026Flag Varieties: A Geometric Framework for Deep Network Alignment
  6. 2026Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space
  7. 2025U-REPA: Aligning Diffusion U-Nets to ViTs
  8. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers