mlsanityData to 14 Oct 2026

Inductive Biases for Zero-shot Systematic Generalization in Language-informed Reinforcement Learning

arXiv 2501.15270 ↗First posted 25 Jan 2025cs.LG cs.AI cs.CLcited by 1

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Sample efficiency and systematic generalization are two long-standing challenges in reinforcement learning. Previous studies have shown that involving natural language along with other observation modalities can improve generalization and sample efficiency due to its compositional and open-ended nature. However, to transfer these properties of language to the decision-making process, it is necessary to establish a proper language grounding mechanism. One approach to this problem is applying inductive biases to extract fine-grained and informative representations from the observations, which makes them more connectable to the language units. We provide architecture-level inductive biases for modularity and sparsity mainly based on Neural Production Systems (NPS). Alongside NPS, we assign a central role to memory in our architecture. It can be seen as a high-level information aggregator which feeds policy/value heads with comprehensive information and simultaneously guides selective attention in NPS through attentional feedback. Our results in the BabyAI environment suggest that the proposed model's systematic generalization and sample efficiency are improved significantly compared to previous models. An extensive ablation study on variants of the proposed method is conducted, and the effectiveness of each employed technique on generalization, sample efficiency, and training stability is specified.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 61 of its 103 references that are in the map. First posted on arXiv 25 Jan 2025.

Builds on

61 of 103 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  9. 2020Object-Centric Learning with Slot Attentioncited by 876
  10. 2019Solving Rubik's Cube with a Robot Handcited by 734
  11. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  12. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
38 more
  1. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  2. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  3. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  4. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  5. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  6. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  7. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 284
  8. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  9. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 220
  10. 2020The NetHack Learning Environmentcited by 198
  11. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  12. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  13. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  14. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 158
  15. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  16. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  17. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  18. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  19. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  20. 2020Invariant Causal Prediction for Block MDPscited by 105
  21. 2019Compositional generalization through meta sequence-to-sequence learningcited by 99
  22. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 88
  23. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  24. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 81
  25. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  26. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  27. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  28. 2020Thinking Fast and Slow in AIcited by 70
  29. 2020Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Explorationcited by 68
  30. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  31. 2020Learning to Recombine and Resample Data for Compositional Generalizationcited by 65
  32. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  33. 2019Human few-shot learning of compositional instructionscited by 47
  34. 2021Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learningcited by 45
  35. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  36. 2021Fast and Slow Learning of Recurrent Independent Mechanismscited by 36
  37. 2020Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learningcited by 32
  38. 2021A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learningcited by 27

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Score Is Not the Structure: Brain Alignment and Cross-Lingual Transfercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
  2. 2026Demonstrating Generalization Failures via Mixtures of Conditional Policies
  3. 2024REFINE-LM: Mitigating Language Model Stereotypes via Reinforcement Learning
  4. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Training
  5. 2024Large Language Models are Biased Reinforcement Learners
  6. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  7. 2024Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models
  8. 2024Inductive Generalization in Reinforcement Learning from Specifications