mlsanityData to 7 Oct 2026

ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning

arXiv 2510.14176 ↗First posted 16 Oct 2025cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) algorithms are highly sensitive to reward function specification, which remains a central challenge limiting their broad applicability. We present ARM-FM: Automated Reward Machines via Foundation Models, a framework for automated, compositional reward design in RL that leverages the high-level reasoning capabilities of foundation models (FMs). Reward machines (RMs) -- an automata-based formalism for reward specification -- are used as the mechanism for RL objective specification, and are automatically constructed via the use of FMs. The structured formalism of RMs yields effective task decompositions, while the use of FMs enables objective specifications in natural language. Concretely, we (i) use FMs to automatically generate RMs from natural language specifications; (ii) associate language embeddings with each RM automata-state to enable generalization across tasks; and (iii) provide empirical evidence of ARM-FM's effectiveness in a diverse suite of challenging environments, including evidence of zero-shot generalization.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 43 of its 62 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

43 of 62 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2025Gemma 3 Technical Reportcited by 1,686
  11. 2024A Survey on LLM-as-a-Judgecited by 1,384
  12. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
31 more
  1. 2017Hindsight Experience Replaycited by 1,096
  2. 2018Exploration by Random Network Distillationcited by 966
  3. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  5. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  6. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  7. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  8. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  9. 2017Count-Based Exploration with Neural Density Modelscited by 255
  10. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  11. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
  12. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  13. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  14. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  15. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  16. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  17. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
  18. 2019Joint Inference of Reward Machines and Policies for Reinforcement Learningcited by 56
  19. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  20. 2021Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logiccited by 38
  21. 2025Meta-World+: An Improved, Standardized, RL Benchmarkcited by 37
  22. 2020Reward Machines for Cooperative Multi-Agent Reinforcement Learningcited by 35
  23. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 28
  24. 2024MaestroMotif: Skill Design from Artificial Intelligence Feedbackcited by 24
  25. 2025Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learningcited by 20
  26. 2024Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learningcited by 18
  27. 2024Reward Machines for Deep RL in Noisy and Uncertain Environmentscited by 11
  28. 2024RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learningcited by 10
  29. 2024Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedbackcited by 10
  30. 2024Using Large Language Models to Automate and Expedite Reinforcement Learning with Reward Machinecited by 9
  31. 2025Pushdown Reward Machines for Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Joint Learning of Hierarchical Neural Options and Abstract World Modelcited by 1
  2. 2026Foundation World Models for Agents that Learn, Verify, and Adapt Reliably Beyond Static Environmentscited by 1
  3. 2026Agentick: A Unified Benchmark for General Sequential Decision-Making Agentscited by 1
  4. 2026The Alignment Flywheel: A Governance-Centric Hybrid MAS for Architecture-Agnostic Safetycited by 0
  5. 2026LANTERN: LLM-Augmented Neurosymbolic Transfer with Experience-Gated Reasoning Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning
  2. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  3. 2026The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination
  4. 2025RM-R1: Reward Modeling as Reasoning
  5. 2025RLAF: Reinforcement Learning from Automaton Feedback
  6. 2026RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models
  7. 2024OCALM: Object-Centric Assessment with Language Models
  8. 2025Reward Evolution with Graph-of-Thoughts: A Bi-Level Language Model Framework for Reinforcement Learning