# Improve Deep Forest with Learnable Layerwise Augmentation Policy Schedule

arXiv: https://arxiv.org/abs/2309.09030 · HTML: https://mlsanity.com/paper/2309.09030/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 16 Sep 2023.

## Builds on (most cited first, 18 of 18)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules](https://mlsanity.com/paper/1905.05393/) (2019)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [DANets: Deep Abstract Networks for Tabular Data Classification and Regression](https://mlsanity.com/paper/2112.02962/) (2021)
- [How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization](https://mlsanity.com/paper/2210.06441/) (2022)
- [Multi-Layered Gradient Boosting Decision Trees](https://mlsanity.com/paper/1806.00007/) (2018)
- [Multi-Level Deep Cascade Trees for Conversion Rate Prediction in Recommendation System](https://mlsanity.com/paper/1805.09484/) (2018)
- [SketchBoost: Fast Gradient Boosted Decision Tree for Multioutput Problems](https://mlsanity.com/paper/2211.12858/) (2022)
- [Analyzing the tree-layer structure of Deep Forests](https://mlsanity.com/paper/2010.15690/) (2020)
- [Data Augmentation for Compositional Data: Advancing Predictive Models of the Microbiome](https://mlsanity.com/paper/2205.09906/) (2022)
- [A Survey of Historical Learning: Learning Models with Learning History](https://mlsanity.com/paper/2303.12992/) (2023)

## Built on by (most cited first, 3 of 3)

- [A Margin-Maximizing Fine-Grained Ensemble Method](https://mlsanity.com/paper/2409.12849/) (2024)
- [RGAR: Recurrence Generation-augmented Retrieval for Factual-aware Medical Question Answering](https://mlsanity.com/paper/2502.13361/) (2025)
- [Efficient and Effective Model Extraction](https://mlsanity.com/paper/2409.14122/) (2024)

## Abstract (arXiv)

As a modern ensemble technique, Deep Forest (DF) employs a cascading structure to construct deep models, providing stronger representational power compared to traditional decision forests. However, its greedy multi-layer learning procedure is prone to overfitting, limiting model effectiveness and generalizability. This paper presents an optimized Deep Forest, featuring learnable, layerwise data augmentation policy schedules. Specifically, We introduce the Cut Mix for Tabular data (CMT) augmentation technique to mitigate overfitting and develop a population-based search algorithm to tailor augmentation intensity for each layer. Additionally, we propose to incorporate outputs from intermediate layers into a checkpoint ensemble for more stable performance. Experimental results show that our method sets new state-of-the-art (SOTA) benchmarks in various tabular classification tasks, outperforming shallow tree ensembles, deep forests, deep neural network, and AutoML competitors. The learned policies also transfer effectively to Deep Forest variants, underscoring its potential for enhancing non-differentiable deep learning modules in tabular signal processing.