# Leveraging class abstraction for commonsense reinforcement learning via residual policy gradient methods

arXiv: https://arxiv.org/abs/2201.12126 · HTML: https://mlsanity.com/paper/2201.12126/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 28 Jan 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [Invariant Causal Prediction for Block MDPs](https://mlsanity.com/paper/2003.06016/) (2020)
- [Deep Reinforcement Learning with a Natural Language Action Space](https://mlsanity.com/paper/1511.04636/) (2015)
- [LTL2Action: Generalizing LTL Instructions for Multi-Task RL](https://mlsanity.com/paper/2102.06858/) (2021)
- [Towards Robust Bisimulation Metric Learning](https://mlsanity.com/paper/2110.14096/) (2021)
- [CSKG: The CommonSense Knowledge Graph](https://mlsanity.com/paper/2012.11490/) (2020)
- [Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines](https://mlsanity.com/paper/2010.03790/) (2020)
- [A Boxology of Design Patterns for Hybrid Learning and Reasoning Systems](https://mlsanity.com/paper/1905.12389/) (2019)
- [ConceptNet at SemEval-2017 Task 2: Extending Word Embeddings with Multilingual Relational Knowledge](https://mlsanity.com/paper/1704.03560/) (2017)
- [WordCraft: An Environment for Benchmarking Commonsense Agents](https://mlsanity.com/paper/2007.09185/) (2020)
- [Contextualize Knowledge Bases with Transformer for End-to-end Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2010.05740/) (2020)
- [Natural Language Processing with Commonsense Knowledge: A Survey](https://mlsanity.com/paper/2108.04674/) (2021)

## Built on by (most cited first, 2 of 2)

- [Knowledge-enhanced Agents for Interactive Text Games](https://mlsanity.com/paper/2305.05091/) (2023)
- [Transferring Procedural Knowledge across Commonsense Tasks](https://mlsanity.com/paper/2304.13867/) (2023)

## Abstract (arXiv)

Enabling reinforcement learning (RL) agents to leverage a knowledge base while learning from experience promises to advance RL in knowledge intensive domains. However, it has proven difficult to leverage knowledge that is not manually tailored to the environment. We propose to use the subclass relationships present in open-source knowledge graphs to abstract away from specific objects. We develop a residual policy gradient method that is able to integrate knowledge across different abstraction levels in the class hierarchy. Our method results in improved sample efficiency and generalisation to unseen objects in commonsense games, but we also investigate failure modes, such as excessive noise in the extracted class knowledge or environments with little class structure.