mlsanityData to 7 Oct 2026

Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments

arXiv 2501.10893 ↗First posted 18 Jan 2025cs.LG cs.AIcited by 93

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Autonomous agents powered by large language models (LLMs) have the potential to enhance human capabilities, assisting with digital tasks from sending emails to performing data analysis. The abilities of existing LLMs at such tasks are often hindered by the lack of high-quality agent data from the corresponding environments they interact with. We propose Learn-by-interact, a data-centric framework to adapt LLM agents to any given environments without human annotations. Learn-by-interact synthesizes trajectories of agent-environment interactions based on documentations, and constructs instructions by summarizing or abstracting the interaction histories, a process called backward construction. We assess the quality of our synthetic data by using them in both training-based scenarios and training-free in-context learning (ICL), where we craft innovative retrieval approaches optimized for agents. Extensive experiments on SWE-bench, WebArena, OSWorld and Spider2-V spanning across realistic coding, web, and desktop environments show the effectiveness of Learn-by-interact in various downstream agentic tasks -- baseline results are improved by up to 12.2% for ICL with Claude-3.5 and 19.5% for training with Codestral-22B. We further demonstrate the critical role of backward construction, which provides up to 14.0% improvement for training. Our ablation studies demonstrate the efficiency provided by our synthesized data in ICL and the superiority of our retrieval pipeline over alternative approaches like conventional retrieval-augmented generation (RAG). We expect that Learn-by-interact will serve as a foundation for agent data synthesis as LLMs are increasingly deployed at real-world environments.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 35 of its 52 references that are in the map. First posted on arXiv 18 Jan 2025.

Builds on

35 of 52 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation1 Aug
  2. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models19 Mar
  3. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agents9 Nov
  4. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMs19 Oct
  5. 2023OpenAgents: An Open Platform for Language Agents in the Wild16 Oct
  6. 2023FireAct: Toward Language Agent Fine-tuning9 Oct
  7. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models6 Oct
  8. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents3 Feb

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  10. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  11. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  12. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
15 more
  1. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  2. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  3. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  4. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  5. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  6. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  7. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  8. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  9. 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 89
  10. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  11. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  12. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 55
  13. 2023Summarization is (Almost) Deadcited by 43
  14. 2024Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Modelcited by 19
  15. 2022Physical Interaction and Manipulation of the Environment using Aerial Robotscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 64; 2026: 29.

93 papers.

Sort:

By area:

  1. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 253
  2. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  3. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
  4. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  5. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  6. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  7. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  8. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 106
  9. 2025Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solvingcited by 90
  10. 2025Memp: Exploring Agent Procedural Memorycited by 88
  11. 2025Agent Learning via Early Experiencecited by 79
  12. 2025Self-Challenging Language Model Agentscited by 56
38 more
  1. 2025Towards General Agentic Intelligence via Environment Scalingcited by 51
  2. 2025Contextual Experience Replay for Self-Improvement of Language Agentscited by 46
  3. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websitescited by 43
  4. 2025AgentSynth: Scalable Task Generation for Generalist Computer-Use Agentscited by 41
  5. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  6. 2025Scaling Agent Learning via Experience Synthesiscited by 40
  7. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  8. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  9. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
  10. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  11. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scalingcited by 24
  12. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 22
  13. 2025VideoAgentTrek: Computer Use Pretraining from Unlabeled Videoscited by 22
  14. 2025ProgRM: Build Better GUI Agents with Progress Rewardscited by 21
  15. 2026Reinforcement World Model Learning for LLM-based Agentscited by 21
  16. 2025WebGuard: Building a Generalizable Guardrail for Web Agentscited by 20
  17. 2025AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learningcited by 18
  18. 2025SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraintscited by 17
  19. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  20. 2025Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarioscited by 14
  21. 2025Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agentscited by 14
  22. 2025AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesiscited by 12
  23. 2025SynthAgent: Adapting Web Agents with Synthetic Supervisioncited by 12
  24. 2025Automated Skill Discovery for Language Agents through Exploration and Iterative Feedbackcited by 11
  25. 2025MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Userscited by 10
  26. 2025Context manipulation attacks : Web agents are susceptible to corrupted memorycited by 10
  27. 2025Secure and Efficient Access Control for Computer-Use Agents via Context Spacecited by 10
  28. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agentscited by 10
  29. 2025OpenApps: Simulating Environment Variations to Measure UI-Agent Reliabilitycited by 10
  30. 2025AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Deliverycited by 9
  31. 2025Watch and Learn: Learning to Use Computers from Online Videoscited by 9
  32. 2025Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systemscited by 8
  33. 2026OpAgent: Operator Agent for Web Navigationcited by 8
  34. 2025Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphscited by 7
  35. 2025WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph Representationcited by 7
  36. 2025Co-Evolving Agents: Learning from Failures as Hard Negativescited by 7
  37. 2025Rethinking Agent Design: From Top-Down Workflows to Bottom-Up Skill Evolutioncited by 6
  38. 2025LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedbackcited by 6

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Test-Time Adaptation for LLM Agents via Environment Interaction
  2. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environments
  3. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environments
  4. 2026Principles that Guide, Actions that Inform: Agent Evolution via Knowledge Abstraction
  5. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence
  6. 2026Experience Funnel: A State-Policy Alternating Loop for Self-Evolving Agents
  7. 2025Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profiles
  8. 2026Unified Context Evolution for LLM Agents