# Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidance

arXiv: https://arxiv.org/abs/2507.17131 · HTML: https://mlsanity.com/paper/2507.17131/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 23 Jul 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning](https://mlsanity.com/paper/2301.11916/) (2023)
- [GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner](https://mlsanity.com/paper/2304.04779/) (2023)
- [UniGraph: Learning a Unified Cross-Domain Foundation Model for Text-Attributed Graphs](https://mlsanity.com/paper/2402.13630/) (2024)
- [MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research](https://mlsanity.com/paper/2505.19955/) (2025)
- [FlowReasoner: Reinforcing Query-Level Meta-Agents](https://mlsanity.com/paper/2504.15257/) (2025)
- [Efficient Inference for Large Reasoning Models: A Survey](https://mlsanity.com/paper/2503.23077/) (2025)
- [Can Indirect Prompt Injection Attacks Be Detected and Removed?](https://mlsanity.com/paper/2502.16580/) (2025)
- [UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs](https://mlsanity.com/paper/2502.00806/) (2025)
- [FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answering](https://mlsanity.com/paper/2405.13873/) (2024)
- [Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?](https://mlsanity.com/paper/2502.12206/) (2025)
- [LLM A*: Human in the Loop Large Language Models Enabled A* Search for Robotics](https://mlsanity.com/paper/2312.01797/) (2023)
- [Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering](https://mlsanity.com/paper/2410.08085/) (2024)
- [Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction](https://mlsanity.com/paper/2504.20472/) (2025)
- [LLM-based Automated Grading with Human-in-the-Loop](https://mlsanity.com/paper/2504.05239/) (2025)

## Built on by (most cited first, 14 of 14)

- [Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory](https://mlsanity.com/paper/2511.20857/) (2025)
- [Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections](https://mlsanity.com/paper/2602.15654/) (2026)
- [EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems](https://mlsanity.com/paper/2510.13220/) (2025)
- [EvoClinician: A Self-Evolving Agent for Multi-Turn Medical Diagnosis via Test-Time Evolutionary Learning](https://mlsanity.com/paper/2601.22964/) (2026)
- [Think in Games: Learning to Reason in Games via Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2508.21365/) (2025)
- [Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluation](https://mlsanity.com/paper/2601.21464/) (2026)
- [What-If Analysis of Large Language Models: Explore the Game World Using Proactive Thinking](https://mlsanity.com/paper/2509.04791/) (2025)
- [HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization](https://mlsanity.com/paper/2606.26614/) (2026)
- [Do Self-Evolving Skills Generalize to Held-Out Tasks?](https://mlsanity.com/paper/2609.39148/) (2026)
- [From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP](https://mlsanity.com/paper/2605.25226/) (2026)
- [Data Intelligence Agents: Interpreting, Modeling, and Querying Enterprise Data via Autonomous Coding Agents](https://mlsanity.com/paper/2606.19319/) (2026)
- [FRAMES: Guarded and Dual-Objective Skill Evolution for Agents in Policy-Governed Enterprise Workflows](https://mlsanity.com/paper/2608.01772/) (2026)
- [FINSKILLOPS: A Self-Evolving Multi-Agent System for SEC Filing QA](https://mlsanity.com/paper/2609.19680/) (2026)
- [SR-Fraud: An Outcome-Supervised Reflective LLM Agent Framework for Non-Stationary Payment Fraud Detection](https://mlsanity.com/paper/2609.27287/) (2026)

## Abstract (arXiv)

Large language model (LLM) agents often struggle in environments where rules and required domain knowledge frequently change, such as regulatory compliance and user risk screening. Current approaches, like offline fine-tuning and standard prompting, are insufficient because they cannot effectively adapt to new knowledge during actual operation. To address this limitation, we propose the Adaptive Reflective Interactive Agent (ARIA), an LLM agent framework designed specifically to continuously learn updated domain knowledge at test time. ARIA assesses its own uncertainty through structured self-dialogue, proactively identifying knowledge gaps and requesting targeted explanations or corrections from human experts. It then systematically updates an internal, timestamped knowledge repository with provided human guidance, detecting and resolving conflicting or outdated knowledge through comparisons and clarification queries. We evaluate ARIA on the realistic customer due diligence name screening task on TikTok Pay, alongside publicly available dynamic knowledge tasks. Results demonstrate significant improvements in adaptability and accuracy compared to baselines using standard offline fine-tuning and existing self-improving agents. ARIA is deployed within TikTok Pay serving over 150 million monthly active users, confirming its practicality and effectiveness for operational use in rapidly evolving environments.