# Learning Computational Efficient Bots with Costly Features

arXiv: https://arxiv.org/abs/2308.09629 · HTML: https://mlsanity.com/paper/2308.09629/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 18 Aug 2023.

## Placed in

- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022)
- [Memory-based control with recurrent neural networks](https://mlsanity.com/paper/1512.04455/) (2015)
- [Classification with Costly Features using Deep Reinforcement Learning](https://mlsanity.com/paper/1711.07364/) (2017)
- [Deep Reinforcement Learning for Navigation in AAA Video Games](https://mlsanity.com/paper/2011.04764/) (2020)
- [Reinforcement Learning with Efficient Active Feature Acquisition](https://mlsanity.com/paper/2011.00825/) (2020)
- [Deep Reinforcement Learning for Cost-Effective Medical Diagnosis](https://mlsanity.com/paper/2302.10261/) (2023)
- [Opportunistic Learning: Budgeted Cost-Sensitive Learning from Data Streams](https://mlsanity.com/paper/1901.00243/) (2019)
- [Active Reinforcement Learning: Observing Rewards at a Cost](https://mlsanity.com/paper/2011.06709/) (2020)
- [Adaptive Transformers in RL](https://mlsanity.com/paper/2004.03761/) (2020)
- [Sequential Cost-Sensitive Feature Acquisition](https://mlsanity.com/paper/1607.03691/) (2016)
- [Feature Selection Using Reinforcement Learning](https://mlsanity.com/paper/2101.09460/) (2021)
- [Graph augmented Deep Reinforcement Learning in the GameRLand3D environment](https://mlsanity.com/paper/2112.11731/) (2021)
- [Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks](https://mlsanity.com/paper/1706.00046/) (2017)
- [Optimizing the Neural Architecture of Reinforcement Learning Agents](https://mlsanity.com/paper/2011.14632/) (2020)
- [Stochastic Adaptive Neural Architecture Search for Keyword Spotting](https://mlsanity.com/paper/1811.06753/) (2018)
- [Efficient Neural Architecture Search with Performance Prediction](https://mlsanity.com/paper/2108.01854/) (2021)
- [Improving the sample-efficiency of neural architecture search with reinforcement learning](https://mlsanity.com/paper/2110.06751/) (2021)
- [Online Budgeted Learning for Classifier Induction](https://mlsanity.com/paper/1903.05382/) (2019)

## Built on by (most cited first, 2 of 2)

- [A Continual Offline Reinforcement Learning Benchmark for Navigation Tasks](https://mlsanity.com/paper/2506.02883/) (2025)
- [A Benchmark Environment for Offline Reinforcement Learning in Racing Games](https://mlsanity.com/paper/2407.09415/) (2024)

## Abstract (arXiv)

Deep reinforcement learning (DRL) techniques have become increasingly used in various fields for decision-making processes. However, a challenge that often arises is the trade-off between both the computational efficiency of the decision-making process and the ability of the learned agent to solve a particular task. This is particularly critical in real-time settings such as video games where the agent needs to take relevant decisions at a very high frequency, with a very limited inference time. In this work, we propose a generic offline learning approach where the computation cost of the input features is taken into account. We derive the Budgeted Decision Transformer as an extension of the Decision Transformer that incorporates cost constraints to limit its cost at inference. As a result, the model can dynamically choose the best input features at each timestep. We demonstrate the effectiveness of our method on several tasks, including D4RL benchmarks and complex 3D environments similar to those found in video games, and show that it can achieve similar performance while using significantly fewer computational resources compared to classical approaches.