# Evolving LLM-Generated Features for Interpretable Classification

arXiv: https://arxiv.org/abs/2610.03951 · HTML: https://mlsanity.com/paper/2610.03951/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering](https://mlsanity.com/paper/2305.03403/) (2023)
- [Concept Bottleneck Large Language Models](https://mlsanity.com/paper/2412.07992/) (2024)
- [Augmenting Interpretable Models with LLMs during Training](https://mlsanity.com/paper/2209.11799/) (2022)
- [LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers](https://mlsanity.com/paper/2503.14434/) (2025)
- [LLM-based feature generation from text for interpretable machine learning](https://mlsanity.com/paper/2409.07132/) (2024)
- [Large Language Models Engineer Too Many Simple Features For Tabular Data](https://mlsanity.com/paper/2410.17787/) (2024)
- [Towards Achieving Concept Completeness for Textual Concept Bottleneck Models](https://mlsanity.com/paper/2502.11100/) (2025)
- [AutoQual: An LLM Agent for Automated Discovery of Interpretable Features for Review Quality Assessment](https://mlsanity.com/paper/2510.08081/) (2025)
- [Evolutionary Pre-Prompt Optimization for Mathematical Reasoning](https://mlsanity.com/paper/2412.04291/) (2024)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used as classifiers, yet they operate as opaque systems whose decisions are difficult to interpret, which complicates their use in regulated domains such as credit scoring or medical diagnosis. We propose an evolutionary framework that iteratively discovers natural language feature definitions (rubrics) for interpretable classification. An LLM generates candidate binary features, evaluates each sample against them, and the resulting vectors can be used to train a transparent classifier such as logistic regression. The feature set evolves over multiple iterations guided by classification errors, per-class activation rates, and feature ablation scores. We evaluate across three benchmarks, including a credit risk dataset representative of regulated domains, comparing single-shot LLM rubrics, evolved rubrics, and direct zero-shot LLM classification. Evolved features improve over single-shot rubrics by +2.9 pp on average and outperform zero-shot LLM classification on two of three tasks, while providing fully auditable decision logic. On the credit risk task, the zero-shot LLM performs at chance (50.7%) with a strong bias toward a single class, whereas evolved features achieve balanced, interpretable predictions. Crucially, this failure is invisible in aggregate accuracy and surfaces only under per-class auditing. Analysis reveals that evolution is most effective when label boundaries cannot be inferred from category names alone or when the LLM lacks reliable domain-specific reasoning.