# Shaking the foundations: delusions in sequence models for interaction and control

arXiv: https://arxiv.org/abs/2110.10819 · HTML: https://mlsanity.com/paper/2110.10819/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 20 Oct 2021.

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Why is Posterior Sampling Better than Optimism for Reinforcement Learning?](https://mlsanity.com/paper/1607.00215/) (2016)
- [Task-Relevant Adversarial Imitation Learning](https://mlsanity.com/paper/1910.01077/) (2019)
- [Causally Correct Partial Models for Reinforcement Learning](https://mlsanity.com/paper/2002.02836/) (2020)
- [Thompson Sampling is Asymptotically Optimal in General Environments](https://mlsanity.com/paper/1602.07905/) (2016)
- [Compress and Control](https://mlsanity.com/paper/1411.5326/) (2014)
- [Causal Analysis of Agent Behavior for AI Safety](https://mlsanity.com/paper/2103.03938/) (2021)

## Built on by (most cited first, 30 of 62)

- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [MotionLM: Multi-Agent Motion Forecasting as Language Modeling](https://mlsanity.com/paper/2309.16534/) (2023)
- [Language Model Cascades](https://mlsanity.com/paper/2207.10342/) (2022)
- [Meaning without reference in large language models](https://mlsanity.com/paper/2208.02957/) (2022)
- [Eight Things to Know about Large Language Models](https://mlsanity.com/paper/2304.00612/) (2023)
- [You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments](https://mlsanity.com/paper/2205.15967/) (2022)
- [SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models](https://mlsanity.com/paper/2305.19308/) (2023)
- [Understanding Causality with Large Language Models: Feasibility and Opportunities](https://mlsanity.com/paper/2304.05524/) (2023)
- [Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving](https://mlsanity.com/paper/2210.09539/) (2022)
- [Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning](https://mlsanity.com/paper/2112.03763/) (2021)
- [CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models](https://mlsanity.com/paper/2112.11941/) (2021)
- [Sequence Model Imitation Learning with Unobserved Contexts](https://mlsanity.com/paper/2208.02225/) (2022)
- [A Survey on Knowledge Editing of Neural Networks](https://mlsanity.com/paper/2310.19704/) (2023)
- [Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback](https://mlsanity.com/paper/2410.05434/) (2024)
- [Passive learning of active causal strategies in agents and language models](https://mlsanity.com/paper/2305.16183/) (2023)
- [Retrieval-Augmented Decision Transformer: External Memory for In-context RL](https://mlsanity.com/paper/2410.07071/) (2024)
- [General Intelligence Requires Rethinking Exploration](https://mlsanity.com/paper/2211.07819/) (2022)
- [A social path to human-like artificial intelligence](https://mlsanity.com/paper/2405.15815/) (2024)
- [Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs](https://mlsanity.com/paper/2402.08733/) (2024)
- [Fighting Fire with Fire: Avoiding DNN Shortcuts through Priming](https://mlsanity.com/paper/2206.10816/) (2022)
- [Causal Semantic Communication for Digital Twins: A Generalizable Imitation Learning Approach](https://mlsanity.com/paper/2304.12502/) (2023)
- [CAR-DESPOT: Causally-Informed Online POMDP Planning for Robots in Confounded Environments](https://mlsanity.com/paper/2304.06848/) (2023)
- [Emergent Causality and the Foundation of Consciousness](https://mlsanity.com/paper/2302.03189/) (2023)

## Abstract (arXiv)

The recent phenomenal success of language models has reinvigorated machine learning research, and large sequence models such as transformers are being applied to a variety of domains. One important problem class that has remained relatively elusive however is purposeful adaptive behavior. Currently there is a common perception that sequence models "lack the understanding of the cause and effect of their actions" leading them to draw incorrect inferences due to auto-suggestive delusions. In this report we explain where this mismatch originates, and show that it can be resolved by treating actions as causal interventions. Finally, we show that in supervised learning, one can teach a system to condition or intervene on data by training with factual and counterfactual error signals respectively.