No Free Swap: Protocol-Dependent Layer Redundancy in Transformers
Not on the map. Why?
Abstract (arXiv)
When researchers ask whether two transformer layers are "equivalent" for compression, they often conflate distinct tests. Replacement asks whether one layer's map can substitute for another's in place; interchange asks whether two layers approximately commute when their positions are swapped. Both are output-grounded swap-KL probes, but they need not agree: on pretrained transformers the protocol gap can change which layers look safe to prune by several-fold under the same evaluator, especially when replacement distances are high. We measure both protocols across checkpoints and architectures. On a Pythia training trajectory (410M and 1.4B), the replacement-interchange gap grows from initialization to convergence. Under one matched WikiText-2 contract at 8B scale, Qwen3-8B enters a divergent regime: interchange-guided removal is several-fold safer than replacement-guided at the same layer budgets, while Llama-3.1-8B ties the two protocols for pruning cost even though interchange KL is lower, showing metric gaps need not map one-to-one to removal. Before layer removal or merging, score both swap-KLs on the target checkpoint; the diagnostic requires only unlabeled forward passes.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 15 May 2026.
Builds on
22 of 27 references.
References · 22
- 2017Attention Is All You Needcited by 60,361
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
- 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
10 more
- 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
- 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
- 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
- 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
- 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
- 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
- 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
- 2024BlockPruner: Fine-grained Pruning for Large Language Modelscited by 36
- 2024EvoPress: Accurate Dynamic Model Compression via Evolutionary Searchcited by 20
- 2025Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMscited by 9
Built on by
Citing papers per year: 2026: 1.
1 paper.