# LLM-based feature generation from text for interpretable machine learning

arXiv: https://arxiv.org/abs/2409.07132 · HTML: https://mlsanity.com/paper/2409.07132/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 11 Sep 2024.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [AutoGluon-Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Models](https://mlsanity.com/paper/2404.16233/) (2024)
- [SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding](https://mlsanity.com/paper/2209.06638/) (2022)
- [Dynamic and Adaptive Feature Generation with LLM](https://mlsanity.com/paper/2406.03505/) (2024)
- [Breaking the Bank with ChatGPT: Few-Shot Text Classification for Finance](https://mlsanity.com/paper/2308.14634/) (2023)
- [An LLM Feature-based Framework for Dialogue Constructiveness Assessment](https://mlsanity.com/paper/2406.14760/) (2024)
- [Data Smells in Public Datasets](https://mlsanity.com/paper/2203.08007/) (2022)
- [Predicting article quality scores with machine learning: The UK Research Excellence Framework](https://mlsanity.com/paper/2212.05415/) (2022)

## Built on by (most cited first, 13 of 13)

- [LLMRank: Understanding LLM Strengths for Model Routing](https://mlsanity.com/paper/2510.01234/) (2025)
- [Automatic Prompt Optimization for Dataset-Level Feature Discovery](https://mlsanity.com/paper/2601.13922/) (2026)
- [Dementia Through Different Eyes: Explainable Modeling of Human and LLM Perceptions for Early Awareness](https://mlsanity.com/paper/2505.13418/) (2025)
- [Random Rule Forest (RRF): Interpretable and Manageable Ensembles of LLM-Generated Questions for Predicting Success from Unstructured Data](https://mlsanity.com/paper/2505.24622/) (2025)
- [Automatic Construction of Clinical Scoring Systems with LLM Agents](https://mlsanity.com/paper/2601.22324/) (2026)
- [Mitigating Label Bias with Interpretable Rubric Embeddings](https://mlsanity.com/paper/2605.21455/) (2026)
- [Data Quality Rule Generation with LLMs](https://mlsanity.com/paper/2609.06053/) (2026)
- [From Stochastic Answers to Verifiable Reasoning: Interpretable Decision-Making with LLM-Generated Code](https://mlsanity.com/paper/2603.13287/) (2026)
- [Principal Trait Analysis: Towards Deriving "Skills" in Human-AI Collaboration](https://mlsanity.com/paper/2608.11460/) (2026)
- [Evaluating LLM-Generated Rules for Heart Disease Prediction](https://mlsanity.com/paper/2609.13192/) (2026)
- [FakeSpotter: A content and strategy agnostic Viral Misinformation Detection Tool](https://mlsanity.com/paper/2609.19152/) (2026)
- [Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions](https://mlsanity.com/paper/2609.34326/) (2026)
- [Evolving LLM-Generated Features for Interpretable Classification](https://mlsanity.com/paper/2610.03951/) (2026)

## Abstract (arXiv)

Existing text representations such as embeddings and bag-of-words are not suitable for rule learning due to their high dimensionality and absent or questionable feature-level interpretability. This article explores whether large language models (LLMs) could address this by extracting a small number of interpretable features from text. We demonstrate this process on two datasets (CORD-19 and M17+) containing several thousand scientific articles from multiple disciplines and a target being a proxy for research impact. An evaluation based on testing for the statistically significant correlation with research impact has shown that LLama 2-generated features are semantically meaningful. We consequently used these generated features in text classification to predict the binary target variable representing the citation rate for the CORD-19 dataset and the ordinal 5-class target representing an expert-awarded grade in the M17+ dataset. Machine-learning models trained on the LLM-generated features provided similar predictive performance to the state-of-the-art embedding model SciBERT for scientific text. The LLM used only 62 features compared to 768 features in SciBERT embeddings, and these features were directly interpretable, corresponding to notions such as article methodological rigor, novelty, or grammatical correctness. As the final step, we extract a small number of well-interpretable action rules. Consistently competitive results obtained with the same LLM feature set across both thematically diverse datasets show that this approach generalizes across domains.