mlsanityData to 14 Oct 2026

Retrospex: Language Agent Meets Offline Reinforcement Learning Critic

arXiv 2505.11807 ↗First posted 17 May 2025cs.CL cs.AIcited by 12

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Large Language Models (LLMs) possess extensive knowledge and commonsense reasoning capabilities, making them valuable for creating powerful agents. However, existing LLM agent frameworks have not fully utilized past experiences for improvement. This work introduces a new LLM-based agent framework called Retrospex, which addresses this challenge by analyzing past experiences in depth. Unlike previous approaches, Retrospex does not directly integrate experiences into the LLM's context. Instead, it combines the LLM's action likelihood with action values estimated by a Reinforcement Learning (RL) Critic, which is trained on past experiences through an offline ''retrospection'' process. Additionally, Retrospex employs a dynamic action rescoring mechanism that increases the importance of experience-based values for tasks that require more interaction with the environment. We evaluate Retrospex in ScienceWorld, ALFWorld and Webshop environments, demonstrating its advantages over strong, contemporary baselines.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 27 of its 36 references that are in the map. First posted on arXiv 17 May 2025.

Builds on

27 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks27 May

Other references · 26

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  10. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
14 more
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  3. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  4. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  5. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  6. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  7. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  8. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  9. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  10. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 159
  11. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  12. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  13. 2024ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomycited by 23
  14. 2023Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actorscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  2. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 19
  3. 2026Self-evolving LLM agents with in-distribution Optimizationcited by 4
  4. 2026RPMS: Enhancing LLM-Based Embodied Planning through Rule-Augmented Memory Synergycited by 3
  5. 2025Symbolic Intermediaries as a Linguistic-Numerical Interface for LLM-Driven Geometric Reasoningcited by 2
  6. 2026STRIDE: A Self-Reflective Agent Framework for Reliable Automatic Equation Discoverycited by 2
  7. 2026Cross-Environment Neural Reranking for Sample-Efficient Action Selection in Text-Based Agentscited by 1
  8. 2026MetaResearcher: Scaling Deep Research via Self-Reflective Reinforcement Learning in Adversarial Virtual Environmentscited by 1
  9. 2026Agent-G2: Gaussian Guidance for Agentic Reinforcement Learningcited by 1
  10. 2026NaiLIA: Multimodal Nail Design Retrieval Based on Dense Intent Descriptions and Palette Queriescited by 0
  11. 2026LifeMem: Enabling Lifelong Experience Reuse for LLM Agentscited by 0
  12. 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Retrospective Language Agents via Joint Policy Gradient Optimization
  2. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback
  3. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
  4. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods
  5. 2026Shockingly Simple Self-retrospection Improves Agentic Models Without RL
  6. 2024SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling
  7. 2026MAGE: Meta-Reinforcement Learning for Language Agents toward Strategic Exploration and Exploitation
  8. 2026ExpGraph: Model-Agnostic Experience Learning with Graph-Structured Memory for LLM Agents