mlsanityData to 7 Oct 2026

History Compression via Language Models in Reinforcement Learning

arXiv 2205.12258 ↗First posted 24 May 2022cs.LG cs.CL stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

In a partially observable Markov decision process (POMDP), an agent typically uses a representation of the past to approximate the underlying MDP. We propose to utilize a frozen Pretrained Language Transformer (PLT) for history representation and compression to improve sample efficiency. To avoid training of the Transformer, we introduce FrozenHopfield, which automatically associates observations with pretrained token embeddings. To form these associations, a modern Hopfield network stores these token embeddings, which are retrieved by queries that are obtained by a random but fixed projection of observations. Our new method, HELM, enables actor-critic network architectures that contain a pretrained language Transformer for history representation as a memory module. Since a representation of the past need not be learned, HELM is much more sample efficient than competitors. On Minigrid and Procgen environments HELM achieves new state-of-the-art results. Our code is available at https://github.com/ml-jku/helm.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 62 of its 114 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

62 of 114 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2016Layer Normalizationcited by 5,579
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
38 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  11. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  12. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  13. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  14. 2020Hopfield Networks is All You Needcited by 461
  15. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  16. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  17. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  18. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  19. 2016Dense Associative Memory for Pattern Recognitioncited by 230
  20. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  21. 2019Recurrent Independent Mechanismscited by 219
  22. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  23. 2020Prioritized Level Replaycited by 190
  24. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  25. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  26. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  27. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  28. 2020Phasic Policy Gradientcited by 121
  29. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
  30. 2021Skill Induction and Planning with Latent Languagecited by 99
  31. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  32. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  33. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  34. 2017Learning with Latent Languagecited by 75
  35. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  36. 2020Grounded Language Learning Fast and Slowcited by 54
  37. 2020Modern Hopfield Networks and Attention for Immune Repertoire Classificationcited by 52
  38. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 17; 2025: 2; 2026: 6.

39 papers.

Sort:

By area:

  1. 2023Large Language Models as General Pattern Machinescited by 196
  2. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  3. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  4. 2023On Sparse Modern Hopfield Modelcited by 41
  5. 2023Conformal Prediction for Time Series with Modern Hopfield Networkscited by 40
  6. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  7. 2023STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Predictioncited by 35
  8. 2023Pretrained Language Models as Visual Planners for Human Assistancecited by 28
  9. 2023Learning to Modulate pre-trained Models in RLcited by 28
  10. 2023Learning Probabilistic Symmetrization for Architecture Agnostic Equivariancecited by 25
  11. 2024Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learningcited by 22
  12. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
27 more
  1. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  2. 2023Exploring the Promise and Limits of Real-Time Recurrent Learningcited by 19
  3. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  4. 2023Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agentscited by 16
  5. 2024Nonparametric Modern Hopfield Modelscited by 16
  6. 2024NAVIX: Scaling MiniGrid Environments with JAXcited by 16
  7. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 15
  8. 2022Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learningcited by 9
  9. 2024Energy-based Hopfield Boosting for Out-of-Distribution Detectioncited by 9
  10. 2024Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RLcited by 7
  11. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  12. 2024Preference Discerning with LLM-Enhanced Generative Retrievalcited by 6
  13. 2024CoRMF: Criticality-Ordered Recurrent Mean Field Ising Solvercited by 3
  14. 2024From Reward Shaping to Q-Shaping: Achieving Unbiased Learning with LLM-Guided Knowledgecited by 3
  15. 2024Enhancing Q-Learning with Large Language Model Heuristicscited by 2
  16. 2024Contrastive Abstraction for Reinforcement Learningcited by 2
  17. 2025Large Language Model-enhanced Reinforcement Learning for Low-Altitude Economy Networkingcited by 2
  18. 2023Beyond PID Controllers: PPO with Neuralized PID Policy for Proton Beam Intensity Control in Mu2ecited by 1
  19. 2024Simplified priors for Object-Centric Learningcited by 1
  20. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observationscited by 1
  21. 2026HiMeS: Hippocampus-inspired Memory System for Personalized AI Assistantscited by 1
  22. 2026AP-OOD: Attention Pooling for Out-of-Distribution Detectioncited by 1
  23. 2026PriorZero: Bridging Language Priors and World Models for Decision Makingcited by 1
  24. 2024CAREL: Instruction-guided reinforcement learning with cross-modal auxiliary objectivescited by 0
  25. 2026MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidancecited by 0
  26. 2026Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Datacited by 0
  27. 2026Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillationcited by 0