mlsanityData to 7 Oct 2026

Training a Generally Curious Agent

arXiv 2502.17543 ↗First posted 24 Feb 2025cs.LG cs.AI cs.CLcited by 29

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Efficient exploration is essential for intelligent systems interacting with their environment, but existing language models often fall short in scenarios that require strategic information gathering. In this paper, we present Paprika, a fine-tuning approach that enables language models to develop general decision-making capabilities that are not confined to particular environments. By training on synthetic interaction data from different tasks that require diverse strategies, Paprika teaches models to explore and adapt their behavior on a new task based on environment feedback in-context without more gradient updates. Experimental results show that models fine-tuned with Paprika can effectively transfer their learned decision-making capabilities to entirely unseen tasks without additional training. Unlike traditional training, our approach's primary bottleneck lies in sampling useful interaction data instead of model updates. To improve sample efficiency, we propose a curriculum learning strategy that prioritizes sampling trajectories from tasks with high learning potential. These results suggest a promising path towards AI systems that can autonomously solve novel sequential decision-making problems that require interactions with the external world.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 55 of its 92 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

55 of 92 references.

References · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  3. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  7. 2023Large Language Modelscited by 774
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  9. 2024Genie: Generative Interactive Environmentscited by 706
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  11. 2016Deep Exploration via Bootstrapped DQNcited by 518
  12. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  13. 2019Introduction to Multi-Armed Banditscited by 475
  14. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  15. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  16. 2019Self-Supervised Exploration via Disagreementcited by 263
  17. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  18. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  19. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  20. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  21. 2020Prioritized Level Replaycited by 190
  22. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  23. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  24. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  25. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  26. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  27. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  28. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
  29. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  30. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  31. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  32. 2021Replay-Guided Adversarial Environment Designcited by 108
  33. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  34. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  35. 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 78
  36. 2024Can large language models explore in-context?cited by 75
  37. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  38. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 13.

29 papers.

Sort:

By area:

  1. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  2. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  3. 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
  4. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 89
  5. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  6. 2025Can Large Reasoning Models Self-Train?cited by 66
  7. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  8. 2026Maximum Likelihood Reinforcement Learningcited by 29
  9. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 27
  10. 2025Meta-RL Induces Exploration in Language Agentscited by 17
  11. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 17
  12. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 16
17 more
  1. 2025Toward Efficient Exploration by Large Language Model Agentscited by 15
  2. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 10
  3. 2025Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMscited by 9
  4. 2025Polychromic Objectives for Reinforcement Learningcited by 7
  5. 2025ABBEL: Learning Natural-Language Belief States for Memory-Efficient Interactioncited by 7
  6. 2026Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RLcited by 6
  7. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 6
  8. 2025When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Trainingcited by 3
  9. 2026Solving Physics Olympiad via Reinforcement Learning on Physics Simulatorscited by 3
  10. 2026Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agentscited by 2
  11. 2026Amortising Bayesian Experimental Design for Sequential Information Gathering in LLMscited by 2
  12. 2025Benchmarking In-context Experiential Learning Through Repeated Product Recommendationscited by 1
  13. 2026Do Reasoning Models Ask Better Questions? A Formal Information-Theoretic Analysis on Multi-Turn LLM Gamescited by 1
  14. 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documentscited by 1
  15. 2025Towards Monotonic Improvement in In-Context Reinforcement Learningcited by 0
  16. 2026ICPRL: Acquiring Physical Intuition from Interactive Controlcited by 0
  17. 2026DORA Explorer: Improving the Exploration Ability of LLMs Without Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Agent Learning via Early Experience
  2. 2025Don't Just Fine-tune the Agent, Tune the Environment
  3. 2025Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback
  4. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
  5. 2024ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomy
  6. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Training
  7. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptation
  8. 2025Towards General Agentic Intelligence via Environment Scaling