unix-ctf: Procedural Environments for Unix-Competence Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Unix competence is the ability to use shell and operating-system primitives as first-class tools, not merely to write programs through a terminal. Current terminal benchmarks tend to blur this distinction: a solver fluent in Python but weak in Unix can pass a substantial fraction of Terminal-Bench 2.0, while the reverse skill profile is rarely exercised. We make the distinction operational and build a training surface for the Unix component. unix-ctf is a procedural generator of capture-the-flag tasks for shell agents. Each task hides a short token (a flag of the form flag(a3b1c9...)) inside a fresh Linux container using a single Unix feature, and the agent must recover it. Tasks are produced by an LLM-assisted synthesis pipeline that generates candidate hiding techniques, rewrites them into parameterized hide-and-find script pairs, and filters them with a bidirectional contract: the hide script must leave no plaintext trace of the flag on disk, and the find script must recover the flag in a fresh directory. Because the LLM only writes the planting and recovery steps (the container, layout, and grading harness are fixed), the pipeline lands 656 of 750 raw attempts as portable, reusable variants (87.5%). Our reproduction of Endless Terminals' full-container-generation approach lands only 17.4% under the same checks. The 656 variants canonicalize to 155 distinct techniques. Fine-tuning Qwen3-8B with LoRA using GRPO on this surface lifts solve rate from 11.6% to 43.6% on a 15-skill multi-family holdout (n=225), redistributes which InterCode-CTF tasks the model solves, and produces a +33 pp gain in Forensics while reaching 32/100 on InterCode-CTF. These results suggest that Unix competence is separable, trainable, and best evaluated directly rather than folded into programming-through-a-shell.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 27 May 2026.
Builds on
24 of 35 references.
References · 24
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2017Hindsight Experience Replaycited by 1,136
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
- 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
12 more
- 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
- 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
- 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 223
- 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
- 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
- 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 164
- 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 158
- 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 93
- 2025Self-Challenging Language Model Agentscited by 59
- 2026Endless Terminals: Scaling RL Environments for Terminal Agentscited by 54
- 2026On Data Engineering for Scaling LLM Terminal Capabilitiescited by 43
- 2025Training Language Model Agents to Find Vulnerabilities with CTF-Dojocited by 15
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026CARE: Pre-Execution Command Verification for Shell-Executing LLM Agents
- 2026Terminal-World: Scaling Terminal-Agent Environments via Agent Skills
- 2026Recursive Synthesis for Long-Horizon Terminal Tasks
- 2026CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
- 2026ECHO: Terminal Agents Learn World Models for Free
- 2026Toward Scalable Terminal Task Synthesis via Skill Graphs
- 2026FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis
- 2026Terminal Agents: A Survey of AI Agents in Command-Line Environments