# GEM: A Gym for Agentic LLMs arXiv: https://arxiv.org/abs/2510.01051 · HTML: https://mlsanity.com/paper/2510.01051/ ## Key facts - Cited by 23 papers in the map (data through 14 Oct 2026). - Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. - Builds on 29 of its 56 references that are in the map. - First posted on arXiv 1 Oct 2025. ## Placed in - LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/) ## Builds on (most cited first, 29 of 29) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019) - [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024) - [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015) - [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024) - [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025) - [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024) - [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025) - [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025) - [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://mlsanity.com/paper/2312.06585/) (2023) - [OpenSpiel: A Framework for Reinforcement Learning in Games](https://mlsanity.com/paper/1908.09453/) (2019) - [VinePPO: Refining Credit Assignment in RL Training of LLMs](https://mlsanity.com/paper/2410.01679/) (2024) - [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018) - [ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.19470/) (2025) - [SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning](https://mlsanity.com/paper/2509.02479/) (2025) - [A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce](https://mlsanity.com/paper/2504.11343/) (2025) - [REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2505.24760/) (2025) - [Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library](https://mlsanity.com/paper/2506.06122/) (2025) - [SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2506.24119/) (2025) - [LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models](https://mlsanity.com/paper/2311.18232/) (2023) - [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023) - [MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Library](https://mlsanity.com/paper/2210.13708/) (2022) - [Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning](https://mlsanity.com/paper/2402.03046/) (2024) ## Built on by (most cited first, 23 of 23) - [SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent](https://mlsanity.com/paper/2511.16108/) (2025) - [Rethinking the Trust Region in LLM Reinforcement Learning](https://mlsanity.com/paper/2602.04879/) (2026) - [ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents](https://mlsanity.com/paper/2603.18815/) (2026) - [Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs](https://mlsanity.com/paper/2511.19773/) (2025) - [RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback](https://mlsanity.com/paper/2603.08561/) (2026) - [LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth](https://mlsanity.com/paper/2602.07962/) (2026) - [Environment Scaling for Interactive Agentic Experience Collection: A Survey](https://mlsanity.com/paper/2511.09586/) (2025) - [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026) - [Verifiable Process Rewards for Agentic Reasoning](https://mlsanity.com/paper/2605.10325/) (2026) - [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026) - [On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length](https://mlsanity.com/paper/2605.02572/) (2026) - [Gym-V: A Unified Vision Environment System for Agentic Vision Research](https://mlsanity.com/paper/2603.15432/) (2026) - [Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements](https://mlsanity.com/paper/2608.17310/) (2026) - [LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structure](https://mlsanity.com/paper/2606.15306/) (2026) - [CASCADE: Case-Based Continual Adaptation for Large Language Models During Deployment](https://mlsanity.com/paper/2605.06702/) (2026) - [Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation](https://mlsanity.com/paper/2609.04298/) (2026) - [Towards Autonomous Business Intelligence via Data-to-Insight Discovery Agent](https://mlsanity.com/paper/2605.07202/) (2026) - [Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks](https://mlsanity.com/paper/2609.08404/) (2026) - [ProCredit: From Outcome Rewards to Progress Credit in Agentic Reinforcement Learning](https://mlsanity.com/paper/2609.27532/) (2026) - [NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents](https://mlsanity.com/paper/2610.03631/) (2026) - [InvestigationWorlds: An Agentic Environment for Legal Investigation](https://mlsanity.com/paper/2610.04129/) (2026) - [BACAM: Behavior-Aware Continual Agent Merging for Multi-Turn Interaction](https://mlsanity.com/paper/2610.04966/) (2026) - [Sherpa: Teaching LLMs to Teach Adaptively](https://mlsanity.com/paper/2610.08778/) (2026) ## Abstract (arXiv) The training paradigm for large language models (LLMs) is moving from static datasets to experience-based learning, where agents acquire skills via interacting with complex environments. To facilitate this transition we introduce GEM (General Experience Maker), an open-source environment simulator designed for the age of LLMs. Analogous to OpenAI-Gym for traditional reinforcement learning (RL), GEM provides a standardized framework for the environment-agent interface, including asynchronous vectorized execution for high throughput, and flexible wrappers for easy extensibility. GEM also features a diverse suite of environments, robust integrated tools, and single-file example scripts demonstrating using GEM with five popular RL training frameworks. Along with this, we also provide a set of baselines across 24 environments using REINFORCE with Return Batch Normalization (ReBN), which -- unlike GRPO -- is compatible with the full RL setting of dense per-turn rewards and offers better credit assignment. We further conduct apple-to-apple benchmarking of PPO, GRPO and REINFORCE in both single- and multi-turn settings using GEM to shed light on the algorithmic designs. Lastly, GEM also functions as a convenient evaluation toolkit besides a training environment. We hope this framework can help accelerate future agentic LLM research.