mlsanityData to 14 Oct 2026

TabQL: In-Context Q-Learning with Tabular Foundation Models

arXiv 2605.18979 ↗First posted 18 May 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

We propose Tabular Q-Learning (TabQL), a reinforcement learning framework that replaces the conventional parametric Q-network in Deep Q-Learning (DQN) with a tabular foundation model endowed with in-context learning capabilities. The key idea is to represent Q-values through a sequence-to-sequence foundation model operating over a tabularized representation of state-action-Q-value tuples, enabling rapid adaptation from limited online interaction by conditioning on recent experience. TabQL departs from classical DQN by leveraging (i) zero- or few-shot Q-value inference via in-context updates, and (ii) a warm-up phase using standard DQN to bootstrap high-quality context. Particularly, to enhance the context quality, new transitions are generated by executing actions output by TabQL with predicted Q values from DQN. We formalize TabQL, analyze its convergence and sample complexity under mild assumptions, and show that TabQL interpolates between vanilla Q-learning and DQN with in-context learning. Our analysis demonstrates that TabQL achieves improved efficiency compared to DQN by amortizing Bellman updates through in-context learning. Extensive numerical experiments with several benchmarks showcase the effectiveness and efficacy of the proposed TabQL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 53 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

30 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025ICR-RL: Deep Reinforcement Learning via In-Context Regression14 Sep

Other references · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  7. 2022Transformers learn in-context by gradient descentcited by 732
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  9. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  11. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 289
  12. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
17 more
  1. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  2. 2022Online Decision Transformercited by 196
  3. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  4. 2023The Learnability of In-Context Learningcited by 127
  5. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  6. 2021Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysiscited by 92
  7. 2022Tutorial on amortized optimizationcited by 83
  8. 2023Elastic Decision Transformercited by 67
  9. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  10. 2022On the Power of Foundation Modelscited by 37
  11. 2023Hyper-Decision Transformer for Efficient Online Policy Adaptationcited by 33
  12. 2019Combining Q-Learning and Search with Amortized Value Estimatescited by 19
  13. 2020Multi-Objective Optimization of the Textile Manufacturing Process Using Deep-Q-Network Based Multi-Agent Reinforcement Learningcited by 3
  14. 2024BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spacescited by 3
  15. 2020Improved Hoeffding's Lemma and Hoeffding's Tail Boundscited by 2
  16. 2022NARS vs. Reinforcement learning: ONA vs. Q-Learningcited by 1
  17. 2025Bellman operator convergence enhancements in reinforcement learning algorithmscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025In-Context Compositional Q-Learning for Offline Reinforcement Learning
  2. 2024Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning
  3. 2024Time-Scale Separation in Q-Learning: Extending TD(triangle) for Action-Value Function Decomposition
  4. 2025ShiQ: Bringing back Bellman to LLMs
  5. 2024Enhancing Q-Learning with Large Language Model Heuristics
  6. 2025Scalable In-Context Q-Learning
  7. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning