# Knowledge Infused Policy Gradients with Upper Confidence Bound for Relational Bandits

arXiv: https://arxiv.org/abs/2106.13895 · HTML: https://mlsanity.com/paper/2106.13895/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 25 references that are in the map.
- First posted on arXiv 25 Jun 2021.

## Builds on (most cited first, 6 of 6)

- [A Survey on Contextual Multi-armed Bandits](https://mlsanity.com/paper/1508.03326/) (2015)
- [Beyond variance reduction: Understanding the true impact of baselines on policy optimization](https://mlsanity.com/paper/2008.13773/) (2020)
- [Knowledge Infused Policy Gradients for Adaptive Pandemic Control](https://mlsanity.com/paper/2102.06245/) (2021)
- [Relational Boosted Bandits](https://mlsanity.com/paper/2012.09220/) (2020)
- [User Friendly Automatic Construction of Background Knowledge: Mode Construction from ER Diagrams](https://mlsanity.com/paper/1912.07650/) (2019)
- [Non-Parametric Learning of Gaifman Models](https://mlsanity.com/paper/2001.00528/) (2020)

## Built on by (most cited first, 6 of 6)

- [Knowledge-intensive Language Understanding for Explainable AI](https://mlsanity.com/paper/2108.01174/) (2021)
- [KSAT: Knowledge-infused Self Attention Transformer -- Integrating Multiple Domain-Specific Contexts](https://mlsanity.com/paper/2210.04307/) (2022)
- [Demo Alleviate: Demonstrating Artificial Intelligence Enabled Virtual Assistance for Telehealth: The Mental Health Case](https://mlsanity.com/paper/2304.00025/) (2023)
- [Differentially Private Federated Combinatorial Bandits with Constraints](https://mlsanity.com/paper/2206.13192/) (2022)
- [Neurosymbolic Value-Inspired AI (Why, What, and How)](https://mlsanity.com/paper/2312.09928/) (2023)
- [RDR: the Recap, Deliberate, and Respond Method for Enhanced Language Understanding](https://mlsanity.com/paper/2312.09932/) (2023)

## Abstract (arXiv)

Contextual Bandits find important use cases in various real-life scenarios such as online advertising, recommendation systems, healthcare, etc. However, most of the algorithms use flat feature vectors to represent context whereas, in the real world, there is a varying number of objects and relations among them to model in the context. For example, in a music recommendation system, the user context contains what music they listen to, which artists create this music, the artist albums, etc. Adding richer relational context representations also introduces a much larger context space making exploration-exploitation harder. To improve the efficiency of exploration-exploitation knowledge about the context can be infused to guide the exploration-exploitation strategy. Relational context representations allow a natural way for humans to specify knowledge owing to their descriptive nature. We propose an adaptation of Knowledge Infused Policy Gradients to the Contextual Bandit setting and a novel Knowledge Infused Policy Gradients Upper Confidence Bound algorithm and perform an experimental analysis of a simulated music recommendation dataset and various real-life datasets where expert knowledge can drastically reduce the total regret and where it cannot.