mlsanityData to 7 Oct 2026

Can language agents be alternatives to PPO? A Preliminary Empirical Study On OpenAI Gym

arXiv 2312.03290 ↗First posted 6 Dec 2023cs.AI cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

The formidable capacity for zero- or few-shot decision-making in language agents encourages us to pose a compelling question: Can language agents be alternatives to PPO agents in traditional sequential decision-making tasks? To investigate this, we first take environments collected in OpenAI Gym as our testbeds and ground them to textual environments that construct the TextGym simulator. This allows for straightforward and efficient comparisons between PPO agents and language agents, given the widespread adoption of OpenAI Gym. To ensure a fair and effective benchmarking, we introduce 5 levels of scenario for accurate domain-knowledge controlling and a unified RL-inspired framework for language agents. Additionally, we propose an innovative explore-exploit-guided language (EXE) agent to solve tasks within TextGym. Through numerical experiments and ablation studies, we extract valuable insights into the decision-making capabilities of language agents and make a preliminary evaluation of their potential to be alternatives to PPO in classical sequential decision-making problems. This paper sheds light on the performance of language agents and paves the way for future research in this exciting domain. Our code is publicly available at~urlhttps://github.com/mail-ecnu/Text-Gym-Agents.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

41 of 55 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
29 more
  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2016OpenAI Gymcited by 1,684
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  9. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  10. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  13. 2022PAL: Program-aided Language Modelscited by 635
  14. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  15. 2023Augmented Language Models: a Surveycited by 398
  16. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  17. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  18. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  19. 2023Tool Learning with Foundation Modelscited by 302
  20. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  21. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  22. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  23. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  24. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 193
  25. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  26. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  27. 2023MindAgent: Emergent Gaming Interactioncited by 87
  28. 2022You Only Live Once: Single-Life Reinforcement Learningcited by 23
  29. 2023Tachikuma: Understading Complex Interactions with Multi-Character and Novel Objects by Large Language Modelscited by 5

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Natural Language Reinforcement Learningcited by 17
  2. 2024Natural Language Reinforcement Learningcited by 4