# Behavior Cloned Transformers are Neurosymbolic Reasoners

arXiv: https://arxiv.org/abs/2210.07382 · HTML: https://mlsanity.com/paper/2210.07382/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 43 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 34)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [ScienceWorld: Is your Agent Smarter than a 5th Grader?](https://mlsanity.com/paper/2203.07540/) (2022)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [Interactive Fiction Games: A Colossal Adventure](https://mlsanity.com/paper/1909.05398/) (2019)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)
- [Keep CALM and Explore: Language Models for Action Generation in Text-based Games](https://mlsanity.com/paper/2010.02903/) (2020)
- [Evaluating Commonsense in Pre-trained Language Models](https://mlsanity.com/paper/1911.11931/) (2019)
- [Graph Constrained Reinforcement Learning for Natural Language Action Spaces](https://mlsanity.com/paper/2001.08837/) (2020)
- [Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning](https://mlsanity.com/paper/1809.02121/) (2018)
- [Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph](https://mlsanity.com/paper/2009.11692/) (2020)
- [Learning Dynamic Belief Graphs to Generalize on Text-Based Games](https://mlsanity.com/paper/2002.09127/) (2020)
- [PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World](https://mlsanity.com/paper/2106.00188/) (2021)
- [NLProlog: Reasoning with Weak Unification for Question Answering in Natural Language](https://mlsanity.com/paper/1906.06187/) (2019)
- [Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines](https://mlsanity.com/paper/2010.03790/) (2020)
- [Counting to Explore and Generalize in Text-based Games](https://mlsanity.com/paper/1806.11525/) (2018)
- [How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds](https://mlsanity.com/paper/2010.00685/) (2020)
- [Learning Knowledge Graph-based World Models of Textual Environments](https://mlsanity.com/paper/2106.09608/) (2021)
- [Neuro-Symbolic Reinforcement Learning with First-Order Logic](https://mlsanity.com/paper/2110.10963/) (2021)
- [Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games](https://mlsanity.com/paper/2010.11655/) (2020)
- [Reading and Acting while Blindfolded: The Need for Semantics in Text Game Agents](https://mlsanity.com/paper/2103.13552/) (2021)
- [A Systematic Survey of Text Worlds as Embodied Natural Language Environments](https://mlsanity.com/paper/2107.04132/) (2021)
- [A Survey of Text Games for Reinforcement Learning informed by Natural Language](https://mlsanity.com/paper/2109.09478/) (2021)
- [TextWorldExpress: Simulating Text Games at One Million Steps Per Second](https://mlsanity.com/paper/2208.01174/) (2022)

## Built on by (most cited first, 10 of 10)

- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [Neuro-Symbolic AI in 2024: A Systematic Review](https://mlsanity.com/paper/2501.05435/) (2025)
- [Language Guided Exploration for RL Agents in Text Environments](https://mlsanity.com/paper/2403.03141/) (2024)
- [STARLING: Self-supervised Training of Text-based Reinforcement Learning Agent with Large Language Models](https://mlsanity.com/paper/2406.05872/) (2024)
- [MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation](https://mlsanity.com/paper/2212.08607/) (2022)
- [EXPLORER: Exploration-guided Reasoning for Textual Reinforcement Learning](https://mlsanity.com/paper/2403.10692/) (2024)
- [A Mixture-of-Experts Approach to Few-Shot Task Transfer in Open-Ended Text Worlds](https://mlsanity.com/paper/2405.06059/) (2024)
- [Self-Supervised Behavior Cloned Transformers are Path Crawlers for Text Games](https://mlsanity.com/paper/2312.04657/) (2023)
- [On the Effects of Fine-tuning Language Models for Text-Based Reinforcement Learning](https://mlsanity.com/paper/2404.10174/) (2024)

## Abstract (arXiv)

In this work, we explore techniques for augmenting interactive agents with information from symbolic modules, much like humans use tools like calculators and GPS systems to assist with arithmetic and navigation. We test our agent's abilities in text games -- challenging benchmarks for evaluating the multi-step reasoning abilities of game agents in grounded, language-based environments. Our experimental study indicates that injecting the actions from these symbolic modules into the action space of a behavior cloned transformer agent increases performance on four text game benchmarks that test arithmetic, navigation, sorting, and common sense reasoning by an average of 22%, allowing an agent to reach the highest possible performance on unseen games. This action injection technique is easily extended to new agents, environments, and symbolic modules.