# Heterogeneous Multi-Agent Reinforcement Learning with Attention for Cooperative and Scalable Feature Transformation

arXiv: https://arxiv.org/abs/2511.21934 · HTML: https://mlsanity.com/paper/2511.21934/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Reinforcement learning](https://mlsanity.com/paper/2405.10369/) (2024)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Multi-Agent Reinforcement Learning is a Sequence Modeling Problem](https://mlsanity.com/paper/2205.14953/) (2022)
- [Feature Engineering for Predictive Modeling using Reinforcement Learning](https://mlsanity.com/paper/1709.07150/) (2017)
- [The autofeat Python Library for Automated Feature Engineering and Selection](https://mlsanity.com/paper/1901.07329/) (2019)
- [Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.04402/) (2021)
- [Group-wise Reinforcement Feature Generation for Optimal and Explainable Representation Space Reconstruction](https://mlsanity.com/paper/2205.14526/) (2022)
- [Self-optimizing Feature Generation via Categorical Hashing Representation and Hierarchical Reinforcement Crossing](https://mlsanity.com/paper/2309.04612/) (2023)
- [Beyond Discrete Selection: Continuous Embedding Space Optimization for Generative Feature Selection](https://mlsanity.com/paper/2302.13221/) (2023)
- [Traceable Automatic Feature Transformation via Cascading Actor-Critic Agents](https://mlsanity.com/paper/2212.13402/) (2022)
- [Traceable Group-Wise Self-Optimizing Feature Transformation Learning: A Dual Optimization Perspective](https://mlsanity.com/paper/2306.16893/) (2023)
- [Reinforcement-Enhanced Autoregressive Feature Transformation: Gradient-steered Search in Continuous Space for Postfix Expressions](https://mlsanity.com/paper/2309.13618/) (2023)
- [Unsupervised Generative Feature Transformation via Graph Contrastive Pre-training and Multi-objective Fine-tuning](https://mlsanity.com/paper/2405.16879/) (2024)
- [A Survey on Data-Centric AI: Tabular Learning from Reinforcement Learning and Generative AI Perspective](https://mlsanity.com/paper/2502.08828/) (2025)
- [Reinforcement Feature Transformation for Polymer Property Performance Prediction](https://mlsanity.com/paper/2409.15616/) (2024)
- [Continuous Optimization for Feature Selection with Permutation-Invariant Embedding and Policy-Guided Search](https://mlsanity.com/paper/2505.11601/) (2025)
- [Feature Interaction Aware Automated Data Representation Transformation](https://mlsanity.com/paper/2309.17011/) (2023)
- [Enhancing Tabular Data Optimization with a Flexible Graph-based Reinforced Exploration Strategy](https://mlsanity.com/paper/2406.07404/) (2024)
- [Collaborative Multi-Agent Reinforcement Learning for Automated Feature Transformation with Graph-Driven Path Optimization](https://mlsanity.com/paper/2504.17355/) (2025)

## Abstract (arXiv)

Feature transformation enhances downstream task performance by generating informative features through mathematical feature crossing. Despite the advancements in deep learning, feature transformation remains essential for structured data, where deep models often struggle to capture complex feature interactions. Prior literature on automated feature transformation has achieved success but often relies on heuristics or exhaustive searches, leading to inefficient and time-consuming processes. Recent works employ reinforcement learning (RL) to enhance traditional approaches through a more effective trial-and-error way. However, two limitations remain: 1) Dynamic feature expansion during the transformation process, which causes instability and increases the learning complexity for RL agents; 2) Insufficient cooperation and communication between agents, which results in suboptimal feature crossing operations and degraded model performance. To address them, we propose a novel heterogeneous multi-agent RL framework to enable cooperative and scalable feature transformation. The framework comprises three heterogeneous agents, grouped into two types, each designed to select essential features and operations for feature crossing. To enhance communication among these agents, we implement a shared critic mechanism that facilitates information exchange during feature transformation. To handle the dynamically expanding feature space, we tailor multi-head attention-based feature agents to select suitable features for feature crossing. Additionally, we introduce a state encoding technique during the optimization process to stabilize and enhance the learning dynamics of the RL agents, resulting in more robust and reliable transformation policies. Finally, we conduct extensive experiments to validate the effectiveness, efficiency, robustness, and interpretability of our model.