# Online Limited Memory Neural-Linear Bandits with Likelihood Matching

arXiv: https://arxiv.org/abs/2102.03799 · HTML: https://mlsanity.com/paper/2102.03799/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 7 Feb 2021.

## Builds on (most cited first, 16 of 16)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [Neural Thompson Sampling](https://mlsanity.com/paper/2010.00827/) (2020)
- [A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1510.03820/) (2015)
- [The Uncertainty Bellman Equation and Exploration](https://mlsanity.com/paper/1709.05380/) (2017)
- [Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning](https://mlsanity.com/paper/1809.02121/) (2018)
- [Neural Contextual Bandits with Deep Representation and Shallow Exploration](https://mlsanity.com/paper/2012.01780/) (2020)
- [Efficient Exploration through Bayesian Deep Q-Networks](https://mlsanity.com/paper/1802.04412/) (2018)
- [Every Model Learned by Gradient Descent Is Approximately a Kernel Machine](https://mlsanity.com/paper/2012.00152/) (2020)
- [Shallow Updates for Deep Reinforcement Learning](https://mlsanity.com/paper/1705.07461/) (2017)
- [Is a picture worth a thousand words? A Deep Multi-Modal Fusion Architecture for Product Classification in e-commerce](https://mlsanity.com/paper/1611.09534/) (2016)

## Built on by (most cited first, 14 of 14)

- [Neural Contextual Bandits without Regret](https://mlsanity.com/paper/2107.03144/) (2021)
- [Sample-Then-Optimize Batch Neural Thompson Sampling](https://mlsanity.com/paper/2210.06850/) (2022)
- [Federated Neural Bandits](https://mlsanity.com/paper/2205.14309/) (2022)
- [Efficient Online Bayesian Inference for Neural Bandits](https://mlsanity.com/paper/2112.00195/) (2021)
- [Anti-Concentrated Confidence Bonuses for Scalable Exploration](https://mlsanity.com/paper/2110.11202/) (2021)
- [Representation-Driven Reinforcement Learning](https://mlsanity.com/paper/2305.19922/) (2023)
- [Online Clustering of Dueling Bandits](https://mlsanity.com/paper/2502.02079/) (2025)
- [Adaptive Behavioral AI: Reinforcement Learning to Enhance Pharmacy Services](https://mlsanity.com/paper/2408.07647/) (2024)
- [Empirical analysis of representation learning and exploration in neural kernel bandits](https://mlsanity.com/paper/2111.03543/) (2021)
- [The Digital Transformation in Health: How AI Can Improve the Performance of Health Systems](https://mlsanity.com/paper/2409.16098/) (2024)
- [Neural Network-Based Bandit: A Medium Access Control for the IIoT Alarm Scenario](https://mlsanity.com/paper/2407.16877/) (2024)
- [Optimizing HIV Patient Engagement with Reinforcement Learning in Resource-Limited Settings](https://mlsanity.com/paper/2408.07629/) (2024)
- [Adaptive User Journeys in Pharma E-Commerce with Reinforcement Learning: Insights from SwipeRx](https://mlsanity.com/paper/2408.08024/) (2024)
- [Designing an Interpretable Interface for Contextual Bandits](https://mlsanity.com/paper/2409.15143/) (2024)

## Abstract (arXiv)

We study neural-linear bandits for solving problems where em both exploration and representation learning play an important role. Neural-linear bandits harnesses the representation power of Deep Neural Networks (DNNs) and combines it with efficient exploration mechanisms by leveraging uncertainty estimation of the model, designed for linear contextual bandits on top of the last hidden layer. In order to mitigate the problem of representation change during the process, new uncertainty estimations are computed using stored data from an unlimited buffer. Nevertheless, when the amount of stored data is limited, a phenomenon called catastrophic forgetting emerges. To alleviate this, we propose a likelihood matching algorithm that is resilient to catastrophic forgetting and is completely online. We applied our algorithm, Limited Memory Neural-Linear with Likelihood Matching (NeuralLinear-LiM2) on a variety of datasets and observed that our algorithm achieves comparable performance to the unlimited memory approach while exhibits resilience to catastrophic forgetting.