# Automaton Distillation: Neuro-Symbolic Transfer Learning for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2310.19137 · HTML: https://mlsanity.com/paper/2310.19137/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 29 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning](https://mlsanity.com/paper/2010.03950/) (2020)
- [DARLA: Improving Zero-Shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/1707.08475/) (2017)
- [Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders](https://mlsanity.com/paper/1804.10469/) (2018)
- [Neurosymbolic Reinforcement Learning with Formally Verified Exploration](https://mlsanity.com/paper/2009.12612/) (2020)
- [Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation](https://mlsanity.com/paper/2102.05714/) (2021)
- [Symbolic LTLf Synthesis](https://mlsanity.com/paper/1705.08426/) (2017)

## Built on by (most cited first, 4 of 4)

- [CADENT: Gated Hybrid Distillation for Sample-Efficient Transfer in Reinforcement Learning](https://mlsanity.com/paper/2602.02532/) (2026)
- [Bridging the Gap: Representation Spaces in Neuro-Symbolic AI](https://mlsanity.com/paper/2411.04393/) (2024)
- [RLAF: Reinforcement Learning from Automaton Feedback](https://mlsanity.com/paper/2510.15728/) (2025)
- [LANTERN: LLM-Augmented Neurosymbolic Transfer with Experience-Gated Reasoning Networks](https://mlsanity.com/paper/2605.05478/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) is a powerful tool for finding optimal policies in sequential decision processes. However, deep RL methods have two weaknesses: collecting the amount of agent experience required for practical RL problems is prohibitively expensive, and the learned policies exhibit poor generalization on tasks outside the training data distribution. To mitigate these issues, we introduce automaton distillation, a form of neuro-symbolic transfer learning in which Q-value estimates from a teacher are distilled into a low-dimensional representation in the form of an automaton. We then propose methods for generating Q-value estimates where symbolic information is extracted from a teacher's Deep Q-Network (DQN). The resulting Q-value estimates are used to bootstrap learning in the target discrete and continuous environment via a modified DQN and Twin-Delayed Deep Deterministic (TD3) loss function, respectively. We demonstrate that automaton distillation decreases the time required to find optimal policies for various decision tasks in new environments, even in a target environment different in structure from the source environment.