mlsanityData to 14 Oct 2026

State Trace Rationale As Auxiliary Task in Reinforcement Learning

arXiv 2609.36867 ↗First posted 29 Sep 2026cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

We propose STRAT, an auxiliary task that trains deep reinforcement learning (RL) agents to predict a short textual trace of their own state. Inspired by human spatial navigation, the description combines landmark, route, and survey knowledge, tracking the agent's position, inventory, goals, and immediate progress. Environment rules generate this text online without human labelling. Our method adds a single auxiliary head to a standard policy. Across 60 sparse-reward XLand-MiniGrid tasks, STRAT solves complex environments where standard RL fails outright, while compacting state representations and preventing rank collapse. Beyond performance gains, the predicted trace provides a readable account of agent beliefs at every step for no extra cost.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 39 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

30 of 39 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  10. 2017Hindsight Experience Replaycited by 1,136
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  12. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
18 more
  1. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  2. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  3. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  4. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  5. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  6. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  7. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
  8. 2016Learning to Navigate in Complex Environmentscited by 198
  9. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  10. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  11. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  12. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  13. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  14. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 102
  15. 2017Learning with Latent Languagecited by 75
  16. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 55
  17. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  18. 2021Tell me why! Explanations support learning relational and causal structurecited by 36

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
  2. 2026Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards
  3. 2026StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction
  4. 2024Learning Future Representation with Synthetic Observations for Sample-efficient Reinforcement Learning
  5. 2025Reinforcement Learning for Large Model: A Survey
  6. 2025A Survey of State Representation Learning for Deep Reinforcement Learning
  7. 2025Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents
  8. 2026Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning