# OpenTinker: Separating Concerns in Agentic Reinforcement Learning

arXiv: https://arxiv.org/abs/2601.07376 · HTML: https://mlsanity.com/paper/2601.07376/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 4 of its 8 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 4 of 4)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025)
- [Agent Lightning: Train ANY AI Agents with Reinforcement Learning](https://mlsanity.com/paper/2508.03680/) (2025)

## Built on by (most cited first, 3 of 3)

- [Agentic AI Systems Should Be Designed as Marginal Token Allocators](https://mlsanity.com/paper/2605.01214/) (2026)
- [AgentJet: A Distributed Swarm Training Framework for Agentic Reinforcement Learning](https://mlsanity.com/paper/2606.04484/) (2026)
- [JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Models](https://mlsanity.com/paper/2607.16074/) (2026)

## Abstract (arXiv)

We introduce textscOpenTinker, an open infrastructure for training large language model (LLM) agents with many LoRA-backed policies over shared execution resources. Modern agent workloads mix supervised fine-tuning (SFT), online reinforcement learning (RL), rollout generation, validation, and multi-turn environment interaction. In such workloads, LoRA adapters are not static inference artifacts: they are frequently updated policy states whose optimizer state, rollout snapshot, and training data attribution must remain consistent. textscOpenTinker centers the runtime around this policy lifecycle. Users define environments, agents, and learning objectives, while the system manages training clients, rollout samplers, checkpoint handles, and policy-version refresh. The same data path supports SFT and RL by converting trajectories into token sequences with explicit masks: context and environment observations condition the model, while generated action tokens carry supervised weights or RL advantages. This design enables multi-LoRA SFT/RL training in which many users, tasks, or agents can share a base model while keeping adapter updates, checkpoints, and rollout snapshots isolated. We describe the system architecture, the adapter lifecycle, the service-backed snapshot handoff used by the current implementation, the backend contract for mixed-adapter rollout kernels, and the training scheduler that isolates adapter-local gradients and optimizer state. Representative validation tasks exercise single-turn, multi-turn, LoRA, and multi-agent agentic training.