# Neuro-Symbolic Embedding for Short and Effective Feature Selection via Autoregressive Generation

arXiv: https://arxiv.org/abs/2404.17157 · HTML: https://mlsanity.com/paper/2404.17157/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 68 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space](https://mlsanity.com/paper/2004.04092/) (2020)
- [Group-wise Reinforcement Feature Generation for Optimal and Explainable Representation Space Reconstruction](https://mlsanity.com/paper/2205.14526/) (2022)
- [Self-optimizing Feature Generation via Categorical Hashing Representation and Hierarchical Reinforcement Crossing](https://mlsanity.com/paper/2309.04612/) (2023)
- [Beyond Discrete Selection: Continuous Embedding Space Optimization for Generative Feature Selection](https://mlsanity.com/paper/2302.13221/) (2023)
- [Traceable Automatic Feature Transformation via Cascading Actor-Critic Agents](https://mlsanity.com/paper/2212.13402/) (2022)
- [Traceable Group-Wise Self-Optimizing Feature Transformation Learning: A Dual Optimization Perspective](https://mlsanity.com/paper/2306.16893/) (2023)
- [Reinforcement-Enhanced Autoregressive Feature Transformation: Gradient-steered Search in Continuous Space for Postfix Expressions](https://mlsanity.com/paper/2309.13618/) (2023)
- [Feature Selection as Deep Sequential Generative Learning](https://mlsanity.com/paper/2403.03838/) (2024)
- [Efficient Reinforced Feature Selection via Early Stopping Traverse Strategy](https://mlsanity.com/paper/2109.14180/) (2021)
- [Knockoff-Guided Feature Selection via A Single Pre-trained Reinforced Agent](https://mlsanity.com/paper/2403.04015/) (2024)
- [From Artificial Neural Networks to Deep Learning for Music Generation -- History, Concepts and Trends](https://mlsanity.com/paper/2004.03586/) (2020)
- [Finetuning Pretrained Transformers into Variational Autoencoders](https://mlsanity.com/paper/2108.02446/) (2021)
- [SAFS: A Deep Feature Selection Approach for Precision Medicine](https://mlsanity.com/paper/1704.05960/) (2017)
- [Representation Learning with Autoencoders for Electronic Health Records: A Comparative Study](https://mlsanity.com/paper/1801.02961/) (2018)

## Built on by (most cited first, 14 of 14)

- [Topology-aware Reinforcement Feature Space Reconstruction for Graph Data](https://mlsanity.com/paper/2411.05742/) (2024)
- [Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teaming](https://mlsanity.com/paper/2504.21304/) (2025)
- [Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives](https://mlsanity.com/paper/2506.24124/) (2025)
- [LLM-ML Teaming: Integrated Symbolic Decoding and Gradient Search for Valid and Stable Generative Feature Transformation](https://mlsanity.com/paper/2506.09085/) (2025)
- [Bridging the Domain Gap in Equation Distillation with Reinforcement Feedback](https://mlsanity.com/paper/2505.15572/) (2025)
- [Distribution Shift Aware Neural Tabular Learning](https://mlsanity.com/paper/2508.19486/) (2025)
- [DELTA: Variational Disentangled Learning for Privacy-Preserving Data Reprogramming](https://mlsanity.com/paper/2509.00693/) (2025)
- [Permutation-Invariant Representation Learning for Robust and Privacy-Preserving Feature Selection](https://mlsanity.com/paper/2510.05535/) (2025)
- [Enhancing Risk Assessment in Transformers with Loss-at-Risk Functions](https://mlsanity.com/paper/2411.02558/) (2024)
- [Towards Data-Centric AI: A Comprehensive Survey of Traditional, Reinforcement, and Generative Approaches for Tabular Data Transformation](https://mlsanity.com/paper/2501.10555/) (2025)
- [Biological Pathway Guided Gene Selection Through Collaborative Reinforcement Learning](https://mlsanity.com/paper/2505.24155/) (2025)
- [Building a Chinese Medical Dialogue System: Integrating Large-scale Corpora and Novel Models](https://mlsanity.com/paper/2410.03521/) (2024)
- [Multi-Agent Procedural Graph Extraction with Structural and Logical Refinement](https://mlsanity.com/paper/2601.19170/) (2026)
- [An LLM-in-the-loop RL Framework for Bioinformatics Feature Selection](https://mlsanity.com/paper/2610.05600/) (2026)

## Abstract (arXiv)

Feature selection aims to identify the optimal feature subset for enhancing downstream models. Effective feature selection can remove redundant features, save computational resources, accelerate the model learning process, and improve the model overall performance. However, existing works are often time-intensive to identify the effective feature subset within high-dimensional feature spaces. Meanwhile, these methods mainly utilize a single downstream task performance as the selection criterion, leading to the selected subsets that are not only redundant but also lack generalizability. To bridge these gaps, we reformulate feature selection through a neuro-symbolic lens and introduce a novel generative framework aimed at identifying short and effective feature subsets. More specifically, we found that feature ID tokens of the selected subset can be formulated as symbols to reflect the intricate correlations among features. Thus, in this framework, we first create a data collector to automatically collect numerous feature selection samples consisting of feature ID tokens, model performance, and the measurement of feature subset redundancy. Building on the collected data, an encoder-decoder-evaluator learning paradigm is developed to preserve the intelligence of feature selection into a continuous embedding space for efficient search. Within the learned embedding space, we leverage a multi-gradient search algorithm to find more robust and generalized embeddings with the objective of improving model performance and reducing feature subset redundancy. These embeddings are then utilized to reconstruct the feature ID tokens for executing the final feature selection. Ultimately, comprehensive experiments and case studies are conducted to validate the effectiveness of the proposed framework.