# Deep Embedding Forest: Forest-based Serving with Deep Embedding Features

arXiv: https://arxiv.org/abs/1703.05291 · HTML: https://mlsanity.com/paper/1703.05291/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2017.

## Built on by (most cited first, 7 of 7)

- [Mixture of Virtual-Kernel Experts for Multi-Objective User Profile Modeling](https://mlsanity.com/paper/2106.07356/) (2021)
- [A Comparison of Decision Forest Inference Platforms from A Database Perspective](https://mlsanity.com/paper/2302.04430/) (2023)
- [xDeepInt: a hybrid architecture for modeling the vector-wise and bit-wise feature interactions](https://mlsanity.com/paper/2301.01089/) (2023)
- [BCFNet: A Balanced Collaborative Filtering Network with Attention Mechanism](https://mlsanity.com/paper/2103.06105/) (2021)
- [Explainable Enterprise Credit Rating via Deep Feature Crossing Network](https://mlsanity.com/paper/2105.13843/) (2021)
- [Making CNNs Interpretable by Building Dynamic Sequential Decision Forests with Top-down Hierarchy Learning](https://mlsanity.com/paper/2106.02824/) (2021)
- [FedHide: Federated Learning by Hiding in the Neighbors](https://mlsanity.com/paper/2409.07808/) (2024)

## Abstract (arXiv)

Deep Neural Networks (DNN) have demonstrated superior ability to extract high level embedding vectors from low level features. Despite the success, the serving time is still the bottleneck due to expensive run-time computation of multiple layers of dense matrices. GPGPU, FPGA, or ASIC-based serving systems require additional hardware that are not in the mainstream design of most commercial applications. In contrast, tree or forest-based models are widely adopted because of low serving cost, but heavily depend on carefully engineered features. This work proposes a Deep Embedding Forest model that benefits from the best of both worlds. The model consists of a number of embedding layers and a forest/tree layer. The former maps high dimensional (hundreds of thousands to millions) and heterogeneous low-level features to the lower dimensional (thousands) vectors, and the latter ensures fast serving. Built on top of a representative DNN model called Deep Crossing, and two forest/tree-based models including XGBoost and LightGBM, a two-step Deep Embedding Forest algorithm is demonstrated to achieve on-par or slightly better performance as compared with the DNN counterpart, with only a fraction of serving time on conventional hardware. After comparing with a joint optimization algorithm called partial fuzzification, also proposed in this paper, it is concluded that the two-step Deep Embedding Forest has achieved near optimal performance. Experiments based on large scale data sets (up to 1 billion samples) from a major sponsored search engine proves the efficacy of the proposed model.