mlsanityData to 7 Oct 2026

Motif: Intrinsic Motivation from Artificial Intelligence Feedback

arXiv 2310.00166 ↗First posted 29 Sep 2023cs.AI cs.LGcited by 99

Not on the map. Why?

Cited by 27 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Exploring rich environments and evaluating one's actions without prior knowledge is immensely challenging. In this paper, we propose Motif, a general method to interface such prior knowledge from a Large Language Model (LLM) with an agent. Motif is based on the idea of grounding LLMs for decision-making without requiring them to interact with the environment: it elicits preferences from an LLM over pairs of captions to construct an intrinsic reward, which is then used to train agents with reinforcement learning. We evaluate Motif's performance and behavior on the challenging, open-ended and procedurally-generated NetHack game. Surprisingly, by only learning to maximize its intrinsic reward, Motif achieves a higher game score than an algorithm directly trained to maximize the score itself. When combining Motif's intrinsic reward with the environment reward, our method significantly outperforms existing approaches and makes progress on tasks where no advancements have ever been made without demonstrations. Finally, we show that Motif mostly generates intuitive human-aligned behaviors which can be steered easily through prompt modifications, while scaling well with the LLM size and the amount of information given in the prompt.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 68 of its 100 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

68 of 100 references.

References · 68

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
38 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  6. 2018Exploration by Random Network Distillationcited by 966
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  8. 2018Behavioral Cloning from Observationcited by 482
  9. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  10. 2015Illuminating search spaces by mapping elitescited by 406
  11. 2023Language Models can Solve Computer Taskscited by 371
  12. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  13. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  14. 2016Variational Intrinsic Controlcited by 262
  15. 2023Reward Design with Language Modelscited by 257
  16. 2014Robots that can adapt like animalscited by 203
  17. 2020The NetHack Learning Environmentcited by 196
  18. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  19. 2022Defining and Characterizing Reward Hackingcited by 184
  20. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  21. 2018Meta-Gradient Reinforcement Learningcited by 156
  22. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  23. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  24. 2017Quality and Diversity Optimization: A Unifying Modular Frameworkcited by 111
  25. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  26. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  27. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  28. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  29. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 84
  30. 2021On the Expressivity of Markov Rewardcited by 71
  31. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  32. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  33. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  34. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  35. 2022Exploration via Elliptical Episodic Bonusescited by 51
  36. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  37. 2023Towards A Unified Agent with Foundation Modelscited by 46
  38. 2021ELLA: Exploration through Learned Language Abstractioncited by 42

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 40; 2025: 34; 2026: 22.

99 papers.

Sort:

By area:

  1. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
  2. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  3. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  4. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  5. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  6. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  7. 2024Vision Language Models are In-Context Value Learnerscited by 84
  8. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  9. 2025A Self-Improving Coding Agentcited by 58
  10. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
  11. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  12. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
38 more
  1. 2023Large Language Model as a Policy Teacher for Training Reinforcement Learning Agentscited by 30
  2. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  3. 2024Controlling Large Language Model Agents with Entropic Activation Steeringcited by 29
  4. 2024GenRL: Multimodal-foundation world models for generalization in embodied agentscited by 29
  5. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  6. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  7. 2024GAVEL: Generating Games Via Evolution and Language Modelscited by 24
  8. 2024MaestroMotif: Skill Design from Artificial Intelligence Feedbackcited by 24
  9. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
  10. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  11. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
  12. 2024Large Language Models and Video Games: A Preliminary Scoping Reviewcited by 23
  13. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
  14. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedbackcited by 20
  15. 2025LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacracited by 20
  16. 2024Playing NetHack with LLMs: Potential & Limitations as Zero-Shot Agentscited by 17
  17. 2024Policy Improvement using Language Feedback Modelscited by 16
  18. 2025Pok'eChamp: an Expert-level Minimax Language Agentcited by 16
  19. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  20. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  21. 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedbackcited by 14
  22. 2025SCAR: Shapley Credit Assignment for More Efficient RLHFcited by 13
  23. 2025M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Qualitycited by 11
  24. 2024Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedbackcited by 10
  25. 2024On the Importance of Uncertainty in Decision-Making with Large Language Modelscited by 9
  26. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Explorationcited by 9
  27. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9
  28. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedbackcited by 9
  29. 2025SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Modelscited by 9
  30. 2025AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agentscited by 9
  31. 2025FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Makingcited by 8
  32. 2024Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RLcited by 7
  33. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Modelscited by 7
  34. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrationscited by 7
  35. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 7
  36. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 7
  37. 2023ACES: Generating Diverse Programming Puzzles with with Autotelic Generative Modelscited by 6
  38. 2023diff History for Neural Language Agentscited by 6

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data
  2. 2026Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications
  3. 2024From Laws to Motivation: Guiding Exploration through Law-Based Reasoning and Rewards
  4. 2025PCGRLLM: Large Language Model-Driven Reward Design for Procedural Content Generation Reinforcement Learning
  5. 2024The intrinsic motivation of reinforcement and imitation learning for sequential tasks
  6. 2023Intrinsic Motivation in Model-based Reinforcement Learning: A Brief Review
  7. 2018Learning to Understand Goal Specifications by Modelling Reward
  8. 2025A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning