mlsanityData to 14 Oct 2026

Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

arXiv 2506.07976 ↗First posted 9 Jun 2025cs.LG cs.AIcited by 34

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

The current paradigm of test-time scaling relies on generating long reasoning traces ("thinking" more) before producing a response. In agent problems that require interaction, this can be done by generating thinking traces before acting in the world. However, this process does not allow agents to acquire new information from the environment or adapt their behavior over time. In this work, we propose to scale test-time interaction, an untapped dimension of test-time scaling that increases the agent's interaction horizon to enable running rich behaviors such as exploration, backtracking, and dynamic re-planning within a single rollout. To demonstrate the promise of this scaling dimension, we study the domain of web agents. We first show that even prompting-based interaction scaling without any training can improve task success on web benchmarks non-trivially. Building on this, we introduce TTI (Test-Time Interaction), a curriculum-based online reinforcement learning (RL) approach that trains agents by adaptively adjusting their rollout lengths. Using a Gemma 3 12B model, TTI produces state-of-the-art open-source, open-data web agents on WebVoyager and WebArena benchmarks. We further show that TTI enables agents to balance exploration and exploitation adaptively. Our results establish interaction scaling as a powerful, complementary axis to scaling per-step compute, offering new avenues for training adaptive agents.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 76 of its 102 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

76 of 102 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making3 Jun
  2. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning14 Apr
  3. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning2 Oct
  4. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents13 Aug
  5. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improve25 Jul

Other references · 71

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2025Gemma 3 Technical Reportcited by 1,824
  9. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  10. 2025s1: Simple test-time scalingcited by 1,341
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  12. 2017Hindsight Experience Replaycited by 1,136
38 more
  1. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  2. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  4. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  5. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  6. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  7. 2024Agent Workflow Memorycited by 383
  8. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  9. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  10. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  11. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  12. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  13. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  14. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 221
  15. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  16. 2017Teacher-Student Curriculum Learningcited by 210
  17. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 195
  18. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
  19. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  20. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  21. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 161
  22. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  23. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  24. 2024Tree Search for Language Model Agentscited by 151
  25. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  26. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  27. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 130
  28. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 123
  29. 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 114
  30. 2025Inducing Programmatic Skills for Agentic Taskscited by 111
  31. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 101
  32. 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environmentscited by 95
  33. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  34. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 84
  35. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 81
  36. 2024Beyond Browsing: API-Based Web Agentscited by 78
  37. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
  38. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 68

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 20.

34 papers.

Sort:

By area:

  1. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
  2. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  3. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 72
  4. 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 44
  5. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 36
  6. 2025PolySkill: Learning Generalizable Skills Through Polymorphic Abstractioncited by 27
  7. 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Taskscited by 19
  8. 2025SSRL: Self-Search Reinforcement Learningcited by 14
  9. 2025Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learningcited by 12
  10. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 9
  11. 2026OpAgent: Operator Agent for Web Navigationcited by 8
  12. 2026Meta-Reinforcement Learning with Self-Reflection for Agentic Searchcited by 8
22 more
  1. 2026Learning to Ideate for Machine Learning Engineering Agentscited by 7
  2. 2026TIDE: Trajectory-based Diagnostic Evaluation of Test-Time Improvement in LLM Agentscited by 7
  3. 2026TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agentscited by 7
  4. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  5. 2025Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrievalcited by 6
  6. 2026EntWorld: A Holistic Environment and Benchmark for Verifiable Enterprise GUI Agentscited by 6
  7. 2026On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Lengthcited by 6
  8. 2025A Survey on AgentOps: Categorization, Challenges, and Future Directionscited by 4
  9. 2026Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generationcited by 4
  10. 2026Quantifying Overclaiming Propensity in Frontier LLM Agentscited by 4
  11. 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbationscited by 3
  12. 2026Structured Distillation of Web Agent Capabilities Enables Generalizationcited by 3
  13. 2025RECODE: Reasoning Through Code Generation for Visual Question Answeringcited by 2
  14. 2025Becoming Experienced Judges: Selective Test-Time Learning for Evaluatorscited by 2
  15. 2026Decocted Experience Improves Test-Time Inference in LLM Agentscited by 2
  16. 2026AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agentscited by 2
  17. 2026Learning to Learn from Language Feedback with Social Meta-Learningcited by 1
  18. 2026PRO-CUA: Process-Reward Optimization for Computer Use Agentscited by 1
  19. 2026WebChallenger: A Reliable and Efficient Generalist Web Agentcited by 1
  20. 2026Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failurescited by 0
  21. 2026Efficient Test-Time Adaptation through Human-AI Interactioncited by 0
  22. 2026Position: It is Time to Virtualize Foundation Models with a Self-evolving Operating System Layercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Extending Test-Time Scaling: A 3D Perspective with Context, Batch, and Turn
  2. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning
  3. 2026TMAS: Scaling Test-Time Compute via Multi-Agent Synergy
  4. 2026Scalable Environments Drive Generalizable Agents
  5. 2026Elastic Horizon: Discovering the Effective Interaction Frontier in Agentic Reinforcement Learning
  6. 2025Environment Scaling for Interactive Agentic Experience Collection: A Survey
  7. 2026Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
  8. 2026ASTER: Agentic Scaling with Tool-integrated Extended Reasoning