mlsanityData to 7 Oct 2026

Sparse Autoencoders for Hypothesis Generation

arXiv 2502.04382 ↗First posted 5 Feb 2025cs.CL cs.AI cs.CYcited by 24

Not on the map. Why?

Abstract (arXiv)

We describe HypotheSAEs, a general method to hypothesize interpretable relationships between text data (e.g., headlines) and a target variable (e.g., clicks). HypotheSAEs has three steps: (1) train a sparse autoencoder on text embeddings to produce interpretable features describing the data distribution, (2) select features that predict the target variable, and (3) generate a natural language interpretation of each feature (e.g., "mentions being surprised or shocked") using an LLM. Each interpretation serves as a hypothesis about what predicts the target variable. Compared to baselines, our method better identifies reference hypotheses on synthetic datasets (at least +0.06 in F1) and produces more predictive hypotheses on real datasets (~twice as many significant findings), despite requiring 1-2 orders of magnitude less compute than recent LLM-based methods. HypotheSAEs also produces novel discoveries on two well-studied tasks: explaining partisan differences in Congressional speeches and identifying drivers of engagement with online headlines.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 21 of its 55 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

21 of 55 references.

References · 21

  1. 2016Pointer Sentinel Mixture Modelscited by 2,992
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  3. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  4. 2020Concept Bottleneck Modelscited by 1,022
  5. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  6. 2024Scaling and evaluating sparse autoencoderscited by 530
  7. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  8. 2013k-Sparse Autoencoderscited by 209
  9. 2020BERTology Meets Biology: Interpreting Attention in Protein Language Modelscited by 85
  10. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 77
  11. 2024Hypothesis Generation with Large Language Modelscited by 68
  12. 2019Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootingscited by 67
9 more
  1. 2024Concept Bottleneck Large Language Modelscited by 41
  2. 2023Interpretable-by-Design Text Understanding with Iteratively Generated Concept Bottleneckcited by 21
  3. 2024Explaining Datasets in Words: Statistical Models with Natural Language Parameterscited by 21
  4. 2024Disentangling Dense Embeddings with Sparse Autoencoderscited by 19
  5. 2024Bayesian Concept Bottleneck Models with LLM Priorscited by 16
  6. 2022Are Neural Topic Models Broken?cited by 11
  7. 2023Using large language models to promote health equitycited by 6
  8. 2024A large language model-based approach to quantifying the effects of social determinants in liver transplant decisionscited by 4
  9. 2023Clinical Notes Reveal Physician Fatiguecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 11.

24 papers.

Sort:

By area:

  1. 2025Correlated Errors in Large Language Modelscited by 42
  2. 2025HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generationcited by 34
  3. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 34
  4. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  5. 2025Position: Use Sparse Autoencoders to Discover Unknownscited by 17
  6. 2026How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?cited by 11
  7. 2025Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoderscited by 7
  8. 2025From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Modelscited by 5
  9. 2025Measuring Sparse Autoencoder Feature Sensitivitycited by 4
  10. 2025Exploratory Causal Inference in SAEncecited by 4
  11. 2025Ensembling Sparse Autoencoderscited by 3
  12. 2025Sparse Autoencoders, Again?cited by 3
12 more
  1. 2026Accelerating Social Science Research via Agentic Hypothesization and Experimentationcited by 3
  2. 2026Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demandscited by 2
  3. 2025Interestingness First Classifierscited by 1
  4. 2025Can SAEs reveal and mitigate racial biases of LLMs in healthcare?cited by 1
  5. 2026Data-Centric Interpretability for LLM-based Multi-Agent Reinforcement Learningcited by 1
  6. 2026ComplLLM: Fine-tuning LLMs to Discover Complementary Signals for Decision-makingcited by 1
  7. 2026Disentangling Steering Vectorscited by 1
  8. 2026Steered Generation via Gradient-Based Optimization on Sparse Query Featurescited by 0
  9. 2026Position: Prioritize Identifying Structure, Not Complex Models, for Scientific Discoverycited by 0
  10. 2026PairSAE: Mechanistic Interpretability from Pair Representations in Protein Co-Foldingcited by 0
  11. 2026What Makes Something Hard(er)? Explaining Question Difficulty in Natural Languagecited by 0
  12. 2026RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routingcited by 0