# GPT-FT: An Efficient Automated Feature Transformation Using GPT for Sequence Reconstruction and Performance Enhancement

arXiv: https://arxiv.org/abs/2508.20824 · HTML: https://mlsanity.com/paper/2508.20824/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Builds on (most cited first, 16 of 16)

- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Feature Engineering for Predictive Modeling using Reinforcement Learning](https://mlsanity.com/paper/1709.07150/) (2017)
- [Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science](https://mlsanity.com/paper/1603.06212/) (2016)
- [The autofeat Python Library for Automated Feature Engineering and Selection](https://mlsanity.com/paper/1901.07329/) (2019)
- [AutoML to Date and Beyond: Challenges and Opportunities](https://mlsanity.com/paper/2010.10777/) (2020)
- [Group-wise Reinforcement Feature Generation for Optimal and Explainable Representation Space Reconstruction](https://mlsanity.com/paper/2205.14526/) (2022)
- [One button machine for automating feature engineering in relational databases](https://mlsanity.com/paper/1706.00327/) (2017)
- [Traceable Automatic Feature Transformation via Cascading Actor-Critic Agents](https://mlsanity.com/paper/2212.13402/) (2022)
- [Traceable Group-Wise Self-Optimizing Feature Transformation Learning: A Dual Optimization Perspective](https://mlsanity.com/paper/2306.16893/) (2023)
- [Reinforcement-Enhanced Autoregressive Feature Transformation: Gradient-steered Search in Continuous Space for Postfix Expressions](https://mlsanity.com/paper/2309.13618/) (2023)
- [Techniques for Automated Machine Learning](https://mlsanity.com/paper/1907.08908/) (2019)
- [DIFER: Differentiable Automated Feature Engineering](https://mlsanity.com/paper/2010.08784/) (2020)
- [Continuous Optimization for Feature Selection with Permutation-Invariant Embedding and Policy-Guided Search](https://mlsanity.com/paper/2505.11601/) (2025)
- [StyleRec: A Benchmark Dataset for Prompt Recovery in Writing Style Transformation](https://mlsanity.com/paper/2504.04373/) (2025)
- [Secure and Efficient General Matrix Multiplication On Cloud Using Homomorphic Encryption](https://mlsanity.com/paper/2405.02238/) (2024)
- [Iterative Feature Space Optimization through Incremental Adaptive Evaluation](https://mlsanity.com/paper/2501.14889/) (2025)

## Built on by (most cited first, 1 of 1)

- [Permutation-Invariant Representation Learning for Robust and Privacy-Preserving Feature Selection](https://mlsanity.com/paper/2510.05535/) (2025)

## Abstract (arXiv)

Feature transformation plays a critical role in enhancing machine learning model performance by optimizing data representations. Recent state-of-the-art approaches address this task as a continuous embedding optimization problem, converting discrete search into a learnable process. Although effective, these methods often rely on sequential encoder-decoder structures that cause high computational costs and parameter requirements, limiting scalability and efficiency. To address these limitations, we propose a novel framework that accomplishes automated feature transformation through four steps: transformation records collection, embedding space construction with a revised Generative Pre-trained Transformer (GPT) model, gradient-ascent search, and autoregressive reconstruction. In our approach, the revised GPT model serves two primary functions: (a) feature transformation sequence reconstruction and (b) model performance estimation and enhancement for downstream tasks by constructing the embedding space. Such a multi-objective optimization framework reduces parameter size and accelerates transformation processes. Experimental results on benchmark datasets show that the proposed framework matches or exceeds baseline performance, with significant gains in computational efficiency. This work highlights the potential of transformer-based architectures for scalable, high-performance automated feature transformation.