# LLM-ML Teaming: Integrated Symbolic Decoding and Gradient Search for Valid and Stable Generative Feature Transformation

arXiv: https://arxiv.org/abs/2506.09085 · HTML: https://mlsanity.com/paper/2506.09085/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 59 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Builds on (most cited first, 30 of 36)

- [Large Language Models in Finance: A Survey](https://mlsanity.com/paper/2311.10723/) (2023)
- [Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering](https://mlsanity.com/paper/2305.03403/) (2023)
- [ARGS: Alignment as Reward-Guided Search](https://mlsanity.com/paper/2402.01694/) (2024)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [Feature Engineering for Predictive Modeling using Reinforcement Learning](https://mlsanity.com/paper/1709.07150/) (2017)
- [The autofeat Python Library for Automated Feature Engineering and Selection](https://mlsanity.com/paper/1901.07329/) (2019)
- [Aligning Large Language Models with Representation Editing: A Control Perspective](https://mlsanity.com/paper/2406.05954/) (2024)
- [TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling](https://mlsanity.com/paper/2410.16033/) (2024)
- [OpenFE: Automated Feature Generation with Expert-level Performance](https://mlsanity.com/paper/2211.12507/) (2022)
- [Augmenting Interpretable Models with LLMs during Training](https://mlsanity.com/paper/2209.11799/) (2022)
- [Group-wise Reinforcement Feature Generation for Optimal and Explainable Representation Space Reconstruction](https://mlsanity.com/paper/2205.14526/) (2022)
- [Self-optimizing Feature Generation via Categorical Hashing Representation and Hierarchical Reinforcement Crossing](https://mlsanity.com/paper/2309.04612/) (2023)
- [Evolutionary Large Language Model for Automated Feature Transformation](https://mlsanity.com/paper/2405.16203/) (2024)
- [Dynamic and Adaptive Feature Generation with LLM](https://mlsanity.com/paper/2406.03505/) (2024)
- [Traceable Automatic Feature Transformation via Cascading Actor-Critic Agents](https://mlsanity.com/paper/2212.13402/) (2022)
- [Traceable Group-Wise Self-Optimizing Feature Transformation Learning: A Dual Optimization Perspective](https://mlsanity.com/paper/2306.16893/) (2023)
- [LLM-Enhanced User-Item Interactions: Leveraging Edge Information for Optimized Recommendations](https://mlsanity.com/paper/2402.09617/) (2024)
- [Feature Selection as Deep Sequential Generative Learning](https://mlsanity.com/paper/2403.03838/) (2024)
- [Unsupervised Generative Feature Transformation via Graph Contrastive Pre-training and Multi-objective Fine-tuning](https://mlsanity.com/paper/2405.16879/) (2024)
- [Double Correction Framework for Denoising Recommendation](https://mlsanity.com/paper/2405.11272/) (2024)
- [Knockoff-Guided Feature Selection via A Single Pre-trained Reinforced Agent](https://mlsanity.com/paper/2403.04015/) (2024)
- [Neuro-Symbolic Embedding for Short and Effective Feature Selection via Autoregressive Generation](https://mlsanity.com/paper/2404.17157/) (2024)
- [A Survey on Data-Centric AI: Tabular Learning from Reinforcement Learning and Generative AI Perspective](https://mlsanity.com/paper/2502.08828/) (2025)
- [Multimodality Invariant Learning for Multimedia-Based New Item Recommendation](https://mlsanity.com/paper/2405.15783/) (2024)
- [Reinforcement Feature Transformation for Polymer Property Performance Prediction](https://mlsanity.com/paper/2409.15616/) (2024)
- [Sculpting Features from Noise: Reward-Guided Hierarchical Diffusion for Task-Optimal Feature Transformation](https://mlsanity.com/paper/2505.15152/) (2025)
- [Revolutionizing Biomarker Discovery: Leveraging Generative AI for Bio-Knowledge-Embedded Continuous Space Exploration](https://mlsanity.com/paper/2409.15612/) (2024)
- [Topology-aware Reinforcement Feature Space Reconstruction for Graph Data](https://mlsanity.com/paper/2411.05742/) (2024)
- [Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teaming](https://mlsanity.com/paper/2504.21304/) (2025)
- [Large Language Models Engineer Too Many Simple Features For Tabular Data](https://mlsanity.com/paper/2410.17787/) (2024)

## Built on by (most cited first, 5 of 5)

- [Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives](https://mlsanity.com/paper/2506.24124/) (2025)
- [Distribution Shift Aware Neural Tabular Learning](https://mlsanity.com/paper/2508.19486/) (2025)
- [StorageXTuner: An LLM Agent-Driven Automatic Tuning Framework for Heterogeneous Storage Systems](https://mlsanity.com/paper/2510.25017/) (2025)
- [Dataforge: Agentic Platform for Autonomous Data Engineering](https://mlsanity.com/paper/2511.06185/) (2025)
- [Evolving Demonstration Optimization for Chain-of-Thought Feature Transformation](https://mlsanity.com/paper/2603.09987/) (2026)

## Abstract (arXiv)

Feature transformation enhances data representation by deriving new features from the original data. Generative AI offers potential for this task, but faces challenges in stable generation (consistent outputs) and valid generation (error-free sequences). Existing methods--traditional MLs' low validity and LLMs' instability--fail to resolve both. We find that LLMs ensure valid syntax, while ML's gradient-steered search stabilizes performance. To bridge this gap, we propose a teaming framework combining LLMs' symbolic generation with ML's gradient optimization. This framework includes four steps: (1) golden examples generation, aiming to prepare high-quality samples with the ground knowledge of the teacher LLM; (2) feature transformation sequence embedding and search, intending to uncover potentially superior embeddings within the latent space; (3) student LLM feature transformation, aiming to distill knowledge from the teacher LLM; (4) LLM-ML decoder teaming, dedicating to combine ML and the student LLM probabilities for valid and stable generation. The experiments on various datasets show that the teaming policy can achieve 5% improvement in downstream performance while reducing nearly half of the error cases. The results also demonstrate the efficiency and robustness of the teaming policy. Additionally, we also have exciting findings on LLMs' capacity to understand the original data.