# Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections

arXiv: https://arxiv.org/abs/2603.14833 · HTML: https://mlsanity.com/paper/2603.14833/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)
- [Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching](https://mlsanity.com/paper/2311.17030/) (2023)
- [Causal Abstraction in Model Interpretability: A Compact Survey](https://mlsanity.com/paper/2410.20161/) (2024)

## Built on by (most cited first, 4 of 4)

- [Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning](https://mlsanity.com/paper/2607.18130/) (2026)
- [Dual Attention Residuals](https://mlsanity.com/paper/2607.18730/) (2026)
- [The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction](https://mlsanity.com/paper/2610.06653/) (2026)
- [Multi-Stream Residual Computation in Diffusion Language Models](https://mlsanity.com/paper/2610.18407/) (2026)

## Abstract (arXiv)

Multi-stream transformer architectures have recently been proposed as a promising direction for managing representation collapse and the vanishing gradient problem for residual connections, yet their internal mechanisms remain unexplored. In particular, the recently introduced Manifold-Constrained Hyper-Connections (mHC) architecture posits multiple residual streams with constrained interaction, but lacks in-depth mechanistic analysis. We present the first open-source mHC language model (https://huggingface.co/wgpeng/mhc-780m) and analyze the multiple-stream architecture with a suite of representation-level metrics and causal interventions to probe how parallel streams encode and utilize information. Specifically, we introduce a systematic stream ablation-and-rescue framework that enables direct causal comparison of residual streams during inference. Through targeted pairwise interventions and controlled recovery experiments, we distinguish functional redundancy from asymmetric utilization and reveal how information is distributed across streams beyond what is observable from representational similarity alone.