mlsanityData to 14 Oct 2026

CE-Bench: Towards a Reliable Contrastive Evaluation Benchmark of Interpretability of Sparse Autoencoders

arXiv 2509.00691 ↗First posted 31 Aug 2025cs.CLcited by 2

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Sparse autoencoders (SAEs) are a promising approach for uncovering interpretable features in large language models (LLMs). While several automated evaluation methods exist for SAEs, most rely on external LLMs. In this work, we introduce CE-Bench, a novel and lightweight contrastive evaluation benchmark for sparse autoencoders, built on a curated dataset of contrastive story pairs. We conduct comprehensive evaluation studies to validate the effectiveness of our approach. Our results show that CE-Bench reliably measures the interpretability of sparse autoencoders and aligns well with existing benchmarks without requiring an external LLM judge, achieving over 70% Spearman correlation with results in SAEBench. The official implementation and evaluation dataset are open-sourced and publicly available.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 10 of its 16 references that are in the map. First posted on arXiv 31 Aug 2025.

Builds on

10 of 16 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025SAEs Are Good for Steering -- If You Select the Right Features26 May
  2. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability12 Mar
  3. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders28 Jan
  4. 2024Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models31 Jul
  5. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  6. 2024Improving Dictionary Learning with Gated Sparse Autoencoders24 Apr

Other references · 4

  1. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  2. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
  3. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  4. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 143

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Feature Rivalry in Sparse Autoencoder Representations: A Mechanistic Study of Uncertainty-Driven Feature Competition in LLMscited by 0
  2. 2026Size Doesn't Matter: Cosine-Scored Sparse Autoencoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Are Sparse Autoencoder Benchmarks Reliable?
  2. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  3. 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility
  4. 2025AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations
  5. 2025Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability
  6. 2024Scaling and evaluating sparse autoencoders
  7. 2025Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations
  8. 2025Evaluating SAE interpretability without explanations