# Agent Identity Evals: Measuring Agentic Identity arXiv: https://arxiv.org/abs/2507.17257 ยท HTML: https://mlsanity.com/paper/2507.17257/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. - Builds on 49 of its 80 references that are in the map. - First posted on arXiv 23 Jul 2025. ## Placed in - LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/) ## Builds on (most cited first, 30 of 49) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023) - [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023) - [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023) - [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023) - [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020) - [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023) - [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019) - [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023) - [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023) - [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024) - [The Illusion of State in State-Space Models](https://mlsanity.com/paper/2404.08819/) (2024) - [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://mlsanity.com/paper/2503.01935/) (2025) - [CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation](https://mlsanity.com/paper/2401.01275/) (2024) - [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023) - [Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models](https://mlsanity.com/paper/2411.14257/) (2024) - [The Ethics of Advanced AI Assistants](https://mlsanity.com/paper/2404.16244/) (2024) - [CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities](https://mlsanity.com/paper/2503.17332/) (2025) - [AgentSims: An Open-Source Sandbox for Large Language Model Evaluation](https://mlsanity.com/paper/2308.04026/) (2023) - [Evaluating the Robustness of Neural Language Models to Input Perturbations](https://mlsanity.com/paper/2108.12237/) (2021) - [PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks](https://mlsanity.com/paper/2411.00081/) (2024) - [Visibility into AI Agents](https://mlsanity.com/paper/2401.13138/) (2024) - [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022) - [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021) - [From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models](https://mlsanity.com/paper/2404.00906/) (2024) ## Built on by (most cited first, 5 of 5) - [Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanisms](https://mlsanity.com/paper/2605.30169/) (2026) - [Runtime-Independent Persistent Agents: Preserving Identity, Memory, and Code Across Models, Harnesses, and Servers](https://mlsanity.com/paper/2609.00546/) (2026) - [Measuring What Persists: Conditioning Mechanisms and a Geometric Framework for AI Agent Identity](https://mlsanity.com/paper/2606.21843/) (2026) - [The AI Accountability Ecosystem in the Era of Language Models](https://mlsanity.com/paper/2608.12320/) (2026) - [Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agents](https://mlsanity.com/paper/2609.13637/) (2026) ## Abstract (arXiv) Central to agentic capability and trustworthiness of language model agents (LMAs) is the extent they maintain stable, reliable, identity over time. However, LMAs inherit pathologies from large language models (LLMs) (statelessness, stochasticity, sensitivity to prompts and linguistically-intermediation) which can undermine their identifiability, continuity, persistence and consistency. This attrition of identity can erode their reliability, trustworthiness and utility by interfering with their agentic capabilities such as reasoning, planning and action. To address these challenges, we introduce textitagent identity evals (AIE), a rigorous, statistically-driven, empirical framework for measuring the degree to which an LMA system exhibit and maintain their agentic identity over time, including their capabilities, properties and ability to recover from state perturbations. AIE comprises a set of novel metrics which can integrate with other measures of performance, capability and agentic robustness to assist in the design of optimal LMA infrastructure and scaffolding such as memory and tools. We set out formal definitions and methods that can be applied at each stage of the LMA life-cycle, and worked examples of how to apply them.