# Towards Flexible Inference in Sequential Decision Problems via Bidirectional Transformers

arXiv: https://arxiv.org/abs/2204.13326 · HTML: https://mlsanity.com/paper/2204.13326/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Planning to Explore via Self-Supervised World Models](https://mlsanity.com/paper/2005.05960/) (2020)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [RvS: What is Essential for Offline RL via Supervised Learning?](https://mlsanity.com/paper/2112.10751/) (2021)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [TransDreamer: Reinforcement Learning with Transformer World Models](https://mlsanity.com/paper/2202.09481/) (2022)
- [Generalized Decision Transformer for Offline Hindsight Information Matching](https://mlsanity.com/paper/2111.10364/) (2021)
- [Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model](https://mlsanity.com/paper/1610.03518/) (2016)
- [PMI-Masking: Principled masking of correlated spans](https://mlsanity.com/paper/2010.01825/) (2020)
- [Preferences Implicit in the State of the World](https://mlsanity.com/paper/1902.04198/) (2019)
- [A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models](https://mlsanity.com/paper/1905.12790/) (2019)
- [Teaching Autoregressive Language Models Complex Tasks By Demonstration](https://mlsanity.com/paper/2109.02102/) (2021)
- [Learning Calibratable Policies using Programmatic Style-Consistency](https://mlsanity.com/paper/1910.01179/) (2019)
- [Assistive Tele-op: Leveraging Transformers to Collect Robotic Task Demonstrations](https://mlsanity.com/paper/2112.05129/) (2021)

## Built on by (most cited first, 6 of 6)

- [Deep Transformer Q-Networks for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2206.01078/) (2022)
- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [Large Sequence Models for Sequential Decision-Making: A Survey](https://mlsanity.com/paper/2306.13945/) (2023)
- [Skill Transformer: A Monolithic Policy for Mobile Manipulation](https://mlsanity.com/paper/2308.09873/) (2023)
- [MobilityDL: A Review of Deep Learning From Trajectory Data](https://mlsanity.com/paper/2402.00732/) (2024)
- [Socialized Learning: A Survey of the Paradigm Shift for Edge Intelligence in Networked Systems](https://mlsanity.com/paper/2404.13348/) (2024)

## Abstract (arXiv)

Randomly masking and predicting word tokens has been a successful approach in pre-training language models for a variety of downstream tasks. In this work, we observe that the same idea also applies naturally to sequential decision making, where many well-studied tasks like behavior cloning, offline RL, inverse dynamics, and waypoint conditioning correspond to different sequence maskings over a sequence of states, actions, and returns. We introduce the FlexiBiT framework, which provides a unified way to specify models which can be trained on many different sequential decision making tasks. We show that a single FlexiBiT model is simultaneously capable of carrying out many tasks with performance similar to or better than specialized models. Additionally, we show that performance can be further improved by fine-tuning our general model on specific tasks of interest.