# C2R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders

arXiv: https://arxiv.org/abs/2606.30609 · HTML: https://mlsanity.com/paper/2606.30609/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 8 of 8)

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Improving Dictionary Learning with Gated Sparse Autoencoders](https://mlsanity.com/paper/2404.16014/) (2024)
- [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024)
- [Understanding Refusal in Language Models with Sparse Autoencoders](https://mlsanity.com/paper/2505.23556/) (2025)
- [Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2410.06981/) (2024)
- [OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features](https://mlsanity.com/paper/2509.22033/) (2025)
- [Resa: Transparent Reasoning Models via SAEs](https://mlsanity.com/paper/2506.09967/) (2025)

## Built on by (most cited first, 1 of 1)

- [When Trees Are Not Enough: Learning Mixed-Topology Feature Graphs with Adaptive Graph Sparse Autoencoders](https://mlsanity.com/paper/2609.36294/) (2026)

## Abstract (arXiv)

Sparse Autoencoders (SAEs) are widely used to interpret large language models by decomposing activations into sparse, human-understandable features, but scaling to large dictionaries exposes fundamental challenges. Systematic studies reveal pervasive feature splitting that fragments coherent concepts into non-atomic latents and widespread feature absorption that creates arbitrary exceptions in general features, severely compromising latent reliability. These issues stem from inconsistent latent assignment across samples: without cross-sample constraints, per-sample optimization often allows a single underlying concept to be inconsistently distributed across multiple redundant or interfering latents. To address this, we introduce C2R (underlineCross-sample underlineConsistency underlineRegularization). C2R explicitly encourages that each semantic feature is consistently represented by a unified latent across the batch by penalizing the co-activation of directionally similar latents. Comprehensive evaluation demonstrates that C2R effectively mitigates both splitting and absorption while, crucially, preserving reconstruction fidelity, providing a principled solution that enhances latent interpretability without degrading model performance. Source code is available at https://github.com/hr-jin/Cross-sample-Consistency-Regularization.