# Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

arXiv: https://arxiv.org/abs/2506.07976 · HTML: https://mlsanity.com/paper/2506.07976/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 76 of its 102 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 30 of 76)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [Agent Workflow Memory](https://mlsanity.com/paper/2409.07429/) (2024)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning](https://mlsanity.com/paper/2410.08146/) (2024)
- [WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning](https://mlsanity.com/paper/2411.02337/) (2024)
- [ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL](https://mlsanity.com/paper/2402.19446/) (2024)
- [Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models](https://mlsanity.com/paper/2502.19417/) (2025)
- [DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning](https://mlsanity.com/paper/2406.11896/) (2024)
- [Teacher-Student Curriculum Learning](https://mlsanity.com/paper/1707.00183/) (2017)
- [Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents](https://mlsanity.com/paper/2408.07199/) (2024)
- [Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks](https://mlsanity.com/paper/2503.09572/) (2025)

## Built on by (most cited first, 30 of 34)

- [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.08755/) (2025)
- [Budget-Aware Tool Use Enables Effective Agent Scaling](https://mlsanity.com/paper/2511.17006/) (2025)
- [VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2509.19803/) (2025)
- [PolySkill: Learning Generalizable Skills Through Polymorphic Abstraction](https://mlsanity.com/paper/2510.15863/) (2025)
- [WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks](https://mlsanity.com/paper/2601.02439/) (2026)
- [SSRL: Self-Search Reinforcement Learning](https://mlsanity.com/paper/2508.10874/) (2025)
- [Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning](https://mlsanity.com/paper/2512.07461/) (2025)
- [Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning](https://mlsanity.com/paper/2510.04786/) (2025)
- [OpAgent: Operator Agent for Web Navigation](https://mlsanity.com/paper/2602.13559/) (2026)
- [Meta-Reinforcement Learning with Self-Reflection for Agentic Search](https://mlsanity.com/paper/2603.11327/) (2026)
- [Learning to Ideate for Machine Learning Engineering Agents](https://mlsanity.com/paper/2601.17596/) (2026)
- [TIDE: Trajectory-based Diagnostic Evaluation of Test-Time Improvement in LLM Agents](https://mlsanity.com/paper/2602.02196/) (2026)
- [TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agents](https://mlsanity.com/paper/2602.11767/) (2026)
- [FutureSim: Replaying World Events to Evaluate Adaptive Agents](https://mlsanity.com/paper/2605.15188/) (2026)
- [Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrieval](https://mlsanity.com/paper/2510.27566/) (2025)
- [EntWorld: A Holistic Environment and Benchmark for Verifiable Enterprise GUI Agents](https://mlsanity.com/paper/2601.17722/) (2026)
- [On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length](https://mlsanity.com/paper/2605.02572/) (2026)
- [A Survey on AgentOps: Categorization, Challenges, and Future Directions](https://mlsanity.com/paper/2508.02121/) (2025)
- [Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generation](https://mlsanity.com/paper/2602.03806/) (2026)
- [Quantifying Overclaiming Propensity in Frontier LLM Agents](https://mlsanity.com/paper/2609.20812/) (2026)
- [Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbations](https://mlsanity.com/paper/2510.00496/) (2025)
- [Structured Distillation of Web Agent Capabilities Enables Generalization](https://mlsanity.com/paper/2604.07776/) (2026)
- [RECODE: Reasoning Through Code Generation for Visual Question Answering](https://mlsanity.com/paper/2510.13756/) (2025)
- [Becoming Experienced Judges: Selective Test-Time Learning for Evaluators](https://mlsanity.com/paper/2512.06751/) (2025)
- [Decocted Experience Improves Test-Time Inference in LLM Agents](https://mlsanity.com/paper/2604.04373/) (2026)
- [AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agents](https://mlsanity.com/paper/2606.05597/) (2026)
- [Learning to Learn from Language Feedback with Social Meta-Learning](https://mlsanity.com/paper/2602.16488/) (2026)
- [PRO-CUA: Process-Reward Optimization for Computer Use Agents](https://mlsanity.com/paper/2605.29119/) (2026)

## Abstract (arXiv)

The current paradigm of test-time scaling relies on generating long reasoning traces ("thinking" more) before producing a response. In agent problems that require interaction, this can be done by generating thinking traces before acting in the world. However, this process does not allow agents to acquire new information from the environment or adapt their behavior over time. In this work, we propose to scale test-time interaction, an untapped dimension of test-time scaling that increases the agent's interaction horizon to enable running rich behaviors such as exploration, backtracking, and dynamic re-planning within a single rollout. To demonstrate the promise of this scaling dimension, we study the domain of web agents. We first show that even prompting-based interaction scaling without any training can improve task success on web benchmarks non-trivially. Building on this, we introduce TTI (Test-Time Interaction), a curriculum-based online reinforcement learning (RL) approach that trains agents by adaptively adjusting their rollout lengths. Using a Gemma 3 12B model, TTI produces state-of-the-art open-source, open-data web agents on WebVoyager and WebArena benchmarks. We further show that TTI enables agents to balance exploration and exploitation adaptively. Our results establish interaction scaling as a powerful, complementary axis to scaling per-step compute, offering new avenues for training adaptive agents.