# What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?

arXiv: https://arxiv.org/abs/2602.01611 · HTML: https://mlsanity.com/paper/2602.01611/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 33 of its 38 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 33)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [AgentTuning: Enabling Generalized Agent Abilities for LLMs](https://mlsanity.com/paper/2310.12823/) (2023)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [WebDancer: Towards Autonomous Information Seeking Agency](https://mlsanity.com/paper/2505.22648/) (2025)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization](https://mlsanity.com/paper/2507.15061/) (2025)
- [Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models](https://mlsanity.com/paper/2403.12881/) (2024)
- [Deep Research: A Survey of Autonomous Research Agents](https://mlsanity.com/paper/2508.12752/) (2025)
- [Overtrained Language Models Are Harder to Fine-Tune](https://mlsanity.com/paper/2503.19206/) (2025)
- [AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories](https://mlsanity.com/paper/2410.07706/) (2024)
- [Towards General Agentic Intelligence via Environment Scaling](https://mlsanity.com/paper/2509.13311/) (2025)
- [AgentRefine: Enhancing Agent Generalization through Refinement Tuning](https://mlsanity.com/paper/2501.01702/) (2025)
- [A Survey on the Optimization of Large Language Model-based Agents](https://mlsanity.com/paper/2503.12434/) (2025)
- [KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models](https://mlsanity.com/paper/2402.15043/) (2024)
- [ATLaS: Agent Tuning via Learning Critical Steps](https://mlsanity.com/paper/2503.02197/) (2025)
- [WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking](https://mlsanity.com/paper/2510.24697/) (2025)
- [Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents](https://mlsanity.com/paper/2510.24702/) (2025)
- [Over-training with Mixup May Hurt Generalization](https://mlsanity.com/paper/2303.01475/) (2023)
- [StackPlanner: A Centralized Hierarchical Multi-Agent System with Task-Experience Memory Management](https://mlsanity.com/paper/2601.05890/) (2026)

## Built on by (most cited first, 6 of 6)

- [Learning CLI Agents with Structured Action Credit under Selective Observation](https://mlsanity.com/paper/2605.08013/) (2026)
- [SPT: Skills as Pre-Training Data for Agentic Language Models](https://mlsanity.com/paper/2608.26563/) (2026)
- [Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AI](https://mlsanity.com/paper/2609.08216/) (2026)
- [Behavior Quotient Learning for Low-Rank Adaptation of LLM Agents](https://mlsanity.com/paper/2609.12896/) (2026)
- [Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI Agents](https://mlsanity.com/paper/2610.01787/) (2026)
- [One Task, Many Interfaces: Simple Cross-Interface Consistency Training Improves LLM Agent Robustness](https://mlsanity.com/paper/2610.20576/) (2026)

## Abstract (arXiv)

Large language models are increasingly evaluated as interactive agents, yet standard agent benchmarks conflate two qualitatively distinct sources of success: semantic tool-use and interface-specific interaction pattern memorization. Because both mechanisms can yield identical task success on the original interface, benchmark scores alone are not identifiable evidence of environment-invariant capability. We propose PIPE, a protocol-level evaluation augmentation for diagnosing interface reliance by minimally rewriting environment interfaces while preserving task semantics and execution behavior. Across 16 environments from AgentBench and AgentGym and a range of open-source and API-based agents, PIPE reveals that trajectory-SFT substantially amplifies interface shortcutting: trained agents degrade sharply under minimal interface rewrites, while non-trajectory-trained models remain largely stable. We further introduce Interface Reliance (IR), a counterbalanced alias-based metric that quantifies preference for training-time interfaces, and show that interface shortcutting exhibits environment-dependent, non-monotonic training dynamics that remain invisible under standard evaluation. Our code is available at https://anonymous.4open.science/r/What-Do-Agents-Learn-from-Trajectory-SFT-Semantics-or-Interfaces--0831/.