# EE-Net: Exploitation-Exploration Neural Networks in Contextual Bandits

arXiv: https://arxiv.org/abs/2110.03177 · HTML: https://mlsanity.com/paper/2110.03177/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 46 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks](https://mlsanity.com/paper/1905.13210/) (2019)
- [Finite-Time Analysis of Kernelised Contextual Bandits](https://mlsanity.com/paper/1309.6869/) (2013)
- [Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling](https://mlsanity.com/paper/1802.09127/) (2018)
- [Neural Thompson Sampling](https://mlsanity.com/paper/2010.00827/) (2020)
- [Neural Contextual Bandits with Deep Representation and Shallow Exploration](https://mlsanity.com/paper/2012.01780/) (2020)
- [Ensemble Sampling](https://mlsanity.com/paper/1705.07347/) (2017)
- [Local Clustering in Contextual Multi-Armed Bandits](https://mlsanity.com/paper/2103.00063/) (2021)
- [Multi-facet Contextual Bandits: A Neural Network Perspective](https://mlsanity.com/paper/2106.03039/) (2021)
- [Generic Outlier Detection in Multi-Armed Bandit](https://mlsanity.com/paper/2007.07293/) (2020)

## Built on by (most cited first, 30 of 44)

- [How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark](https://mlsanity.com/paper/2406.06647/) (2024)
- [VCR-Graphormer: A Mini-batch Graph Transformer via Virtual Connections](https://mlsanity.com/paper/2403.16030/) (2024)
- [Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond](https://mlsanity.com/paper/2403.10667/) (2024)
- [Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuning](https://mlsanity.com/paper/2505.16270/) (2025)
- [Graph Neural Bandits](https://mlsanity.com/paper/2308.10808/) (2023)
- [BACKTIME: Backdoor Attacks on Multivariate Time Series Forecasting](https://mlsanity.com/paper/2410.02195/) (2024)
- [SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence](https://mlsanity.com/paper/2502.08767/) (2025)
- [Federated Neural Bandits](https://mlsanity.com/paper/2205.14309/) (2022)
- [Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization](https://mlsanity.com/paper/2506.17252/) (2025)
- [Communication Efficient Distributed Learning for Kernelized Contextual Bandits](https://mlsanity.com/paper/2206.04835/) (2022)
- [PageRank Bandits for Link Prediction](https://mlsanity.com/paper/2411.01410/) (2024)
- [Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?](https://mlsanity.com/paper/2412.08174/) (2024)
- [Real-Time Aligned Reward Model beyond Semantics](https://mlsanity.com/paper/2601.22664/) (2026)
- [Improved Algorithms for Neural Active Learning](https://mlsanity.com/paper/2210.00423/) (2022)
- [Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2602.08499/) (2026)
- [Neural Contextual Bandits for Personalized Recommendation](https://mlsanity.com/paper/2312.14037/) (2023)
- [Neural Active Learning Beyond Bandits](https://mlsanity.com/paper/2404.12522/) (2024)
- [Joint Optimal Transport and Embedding for Network Alignment](https://mlsanity.com/paper/2502.19334/) (2025)
- [PolicySim: An LLM-Based Agent Social Simulation Sandbox for Proactive Policy Optimization](https://mlsanity.com/paper/2603.19649/) (2026)
- [Neural Bandit with Arm Group Graph](https://mlsanity.com/paper/2206.03644/) (2022)
- [Doubly High-Dimensional Contextual Bandits: An Interpretable Model for Joint Assortment-Pricing](https://mlsanity.com/paper/2309.08634/) (2023)
- [Neural Collaborative Filtering Bandits via Meta Learning](https://mlsanity.com/paper/2201.13395/) (2022)
- [Neural Exploitation and Exploration of Contextual Bandits](https://mlsanity.com/paper/2305.03784/) (2023)
- [Meta Clustering of Neural Bandits](https://mlsanity.com/paper/2408.05586/) (2024)
- [Neural Logistic Bandits](https://mlsanity.com/paper/2505.02069/) (2025)
- [Influence Maximization via Graph Neural Bandits](https://mlsanity.com/paper/2406.12835/) (2024)
- [Active Human Feedback Collection via Neural Contextual Dueling Bandits](https://mlsanity.com/paper/2504.12016/) (2025)
- [Mixture of Sequence: Theme-Aware Mixture-of-Experts for Long-Sequence Recommendation](https://mlsanity.com/paper/2604.20858/) (2026)
- [Variance-Aware Linear UCB with Deep Representation for Neural Contextual Bandits](https://mlsanity.com/paper/2411.05979/) (2024)
- [Online Clustering of Dueling Bandits](https://mlsanity.com/paper/2502.02079/) (2025)

## Abstract (arXiv)

In this paper, we propose a novel neural exploration strategy in contextual bandits, EE-Net, distinct from the standard UCB-based and TS-based approaches. Contextual multi-armed bandits have been studied for decades with various applications. To solve the exploitation-exploration tradeoff in bandits, there are three main techniques: epsilon-greedy, Thompson Sampling (TS), and Upper Confidence Bound (UCB). In recent literature, linear contextual bandits have adopted ridge regression to estimate the reward function and combine it with TS or UCB strategies for exploration. However, this line of works explicitly assumes the reward is based on a linear function of arm vectors, which may not be true in real-world datasets. To overcome this challenge, a series of neural bandit algorithms have been proposed, where a neural network is used to learn the underlying reward function and TS or UCB are adapted for exploration. Instead of calculating a large-deviation based statistical bound for exploration like previous methods, we propose "EE-Net", a novel neural-based exploration strategy. In addition to using a neural network (Exploitation network) to learn the reward function, EE-Net uses another neural network (Exploration network) to adaptively learn potential gains compared to the currently estimated reward for exploration. Then, a decision-maker is constructed to combine the outputs from the Exploitation and Exploration networks. We prove that EE-Net can achieve O(√Tlog T) regret and show that EE-Net outperforms existing linear and neural contextual bandit baselines on real-world datasets.