mlsanityData to 14 Oct 2026

SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewards

arXiv 2602.21158 ↗First posted 24 Feb 2026cs.LG cs.CLcited by 9

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large language models (LLMs) are increasingly deployed as multi-step decision-making agents, where effective reward design is essential for guiding learning. Although recent work explores various forms of reward shaping and step-level credit assignment, a key signal remains largely overlooked: the intrinsic uncertainty of LLMs. Uncertainty reflects model confidence, reveals where exploration is needed, and offers valuable learning cues even in failed trajectories. We introduce SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewards, a reinforcement learning framework that incorporates uncertainty directly into the reward design. SELAUR integrates entropy-, least-confidence-, and margin-based metrics into a combined token-level uncertainty estimate, providing dense confidence-aligned supervision, and employs a failure-aware reward reshaping mechanism that injects these uncertainty signals into step- and trajectory-level rewards to improve exploration efficiency and learning stability. Experiments on two benchmarks, ALFWorld and WebShop, show that our method consistently improves success rates over strong baselines. Ablation studies further demonstrate how uncertainty signals enhance exploration and robustness.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 47 of its 60 references that are in the map. First posted on arXiv 24 Feb 2026.

Builds on

47 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  2. 2024Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting25 Oct
  3. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents4 Mar
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  12. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
30 more
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  4. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  5. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  6. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  7. 2020Selective Question Answering under Domain Shiftcited by 238
  8. 2025Survey on Evaluation of LLM-based Agentscited by 204
  9. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  10. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  11. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  12. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  13. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  14. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documentscited by 61
  15. 2025Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistantcited by 54
  16. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 52
  17. 2025Structured Uncertainty guided Clarification for LLM Agentscited by 41
  18. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
  19. 2024Calibrating Long-form Generations from Large Language Modelscited by 32
  20. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  21. 2024MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertaintycited by 27
  22. 2024Soft Self-Consistency Improves Language Model Agentscited by 25
  23. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 25
  24. 2025CP-Router: An Uncertainty-Aware Router Between LLM and LRMcited by 20
  25. 2024SAUP: Situation Awareness Uncertainty Propagation on LLM Agentcited by 14
  26. 2025QuCo-RAG: Quantifying Uncertainty from the Pre-training Corpus for Dynamic Retrieval-Augmented Generationcited by 12
  27. 2023OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environmentscited by 7
  28. 2025Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Modelscited by 7
  29. 2024Adaptive Reward Design for Reinforcement Learningcited by 6
  30. 2025Robust Uncertainty Quantification for Self-Evolving Large Language Models via Continual Domain Pretrainingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Librariescited by 6
  2. 2026Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learningcited by 4
  3. 2026AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?cited by 4
  4. 2026Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carriercited by 1
  5. 2026From Sequence to Structure: Relational Uncertainty Propagation for LLM Agentscited by 1
  6. 2026Monitoring Web Agents Without Internal Signals: Observable Trajectories and Key-Step Supervisioncited by 1
  7. 2026GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligencecited by 0
  8. 2026Experience Memory Graph: One-Shot Error Correction for Agentscited by 0
  9. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
  2. 2025Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning
  3. 2026From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models
  4. 2026Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning
  5. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration
  6. 2025When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training
  7. 2025Boosting Universal LLM Reward Design through Heuristic Reward Observation Space Evolution
  8. 2026Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning