# Causal Mechanism Reduction: Mechanism Replacement for Neural Network Pruning and Abstraction

arXiv: https://arxiv.org/abs/2602.24266 · HTML: https://mlsanity.com/paper/2602.24266/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 27 Feb 2026.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [Causal Consistency of Structural Equation Models](https://mlsanity.com/paper/1707.00819/) (2017)
- [Inducing Causal Structure for Interpretable Neural Networks](https://mlsanity.com/paper/2112.00826/) (2021)
- [Abstracting Causal Models](https://mlsanity.com/paper/1812.03789/) (2018)
- [Reducing Overfitting in Deep Networks by Decorrelating Representations](https://mlsanity.com/paper/1511.06068/) (2015)
- [BackPACK: Packing more into backprop](https://mlsanity.com/paper/1912.10985/) (2019)
- [X-Pruner: eXplainable Pruning for Vision Transformers](https://mlsanity.com/paper/2303.04935/) (2023)
- [Approximate Causal Abstraction](https://mlsanity.com/paper/1906.11583/) (2019)
- [The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?](https://mlsanity.com/paper/2507.08802/) (2025)
- [Data-Efficient Structured Pruning via Submodular Optimization](https://mlsanity.com/paper/2203.04940/) (2022)
- [Causal Abstraction with Soft Interventions](https://mlsanity.com/paper/2211.12270/) (2022)
- [Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning](https://mlsanity.com/paper/2406.03276/) (2024)

## Built on by (most cited first, 1 of 1)

- [Certified Interventional Fidelity: Anytime-Valid, Adaptive Evaluation of Causal Claims in Mechanistic Interpretability](https://mlsanity.com/paper/2607.08349/) (2026)

## Abstract (arXiv)

Which internal mechanisms of a neural network can be replaced while preserving the computation it performs? Structured pruning asks for smaller deployable networks; causal abstraction asks for high-level models that commute with interventions. We introduce causal mechanism reduction (CMR), a framework that treats a trained network as a deterministic structural causal model and replaces selected internal variables by constants or affine functions of retained variables. These replacements compile exactly into smaller dense networks by bias and weight folding, and induce reduced causal models testable with interchange interventions. We derive a unified second-order replacement-risk objective whose special cases recover mean replacement, variance-based pruning (VBP), logit-distortion scoring, and affine neuron merging, together with a margin-based certificate linking logit distortion to interchange-intervention agreement. The framework also exposes a basic invariance requirement: functionally identical ReLU networks should induce the same reduction. Under exact positive-scaling reparameterizations, VBP's kept set collapses to chance-level overlap while the logit-distortion score is exactly invariant. Empirically, CMR variants are competitive with VBP under matched fine-tuning of DeiT-Tiny on ImageNet-100; the clearer separation appears in the invariance and interchange tests, where the logit-distortion score preserves kept sets and consistently improves distributional fidelity. CMR thus gives pruning, compilation, and causal-abstraction verification a common object to optimize and verify.