mlsanityData to 14 Oct 2026

Agent Identity Evals: Measuring Agentic Identity

arXiv 2507.17257 ↗First posted 23 Jul 2025cs.AI cs.MAcited by 5

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Central to agentic capability and trustworthiness of language model agents (LMAs) is the extent they maintain stable, reliable, identity over time. However, LMAs inherit pathologies from large language models (LLMs) (statelessness, stochasticity, sensitivity to prompts and linguistically-intermediation) which can undermine their identifiability, continuity, persistence and consistency. This attrition of identity can erode their reliability, trustworthiness and utility by interfering with their agentic capabilities such as reasoning, planning and action. To address these challenges, we introduce textitagent identity evals (AIE), a rigorous, statistically-driven, empirical framework for measuring the degree to which an LMA system exhibit and maintain their agentic identity over time, including their capabilities, properties and ability to recover from state perturbations. AIE comprises a set of novel metrics which can integrate with other measures of performance, capability and agentic robustness to assist in the design of optimal LMA infrastructure and scaffolding such as memory and tools. We set out formal definitions and methods that can be applied at each stage of the LMA life-cycle, and worked examples of how to apply them.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 49 of its 80 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

49 of 80 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents3 Mar
  2. 2025Position: Stop Acting Like Language Model Agents Are Normal Agents4 Feb
  3. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Review5 Apr
  4. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug
  6. 2023Harms from Increasingly Agentic Algorithmic Systems20 Feb

Other references · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  9. 2020Object-Centric Learning with Slot Attentioncited by 876
  10. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  11. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  12. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
31 more
  1. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  2. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  3. 2024The Illusion of State in State-Space Modelscited by 187
  4. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluationcited by 143
  5. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 115
  6. 2024The Ethics of Advanced AI Assistantscited by 107
  7. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 96
  8. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 88
  9. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  10. 2024PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Taskscited by 82
  11. 2024Visibility into AI Agentscited by 79
  12. 2022Scene Graph Generation: A Comprehensive Surveycited by 73
  13. 2021Agent Incentives: A Causal Perspectivecited by 51
  14. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  15. 2024Benchmarking Agentic Workflow Generationcited by 37
  16. 2023Towards LLM-driven Dialogue State Trackingcited by 25
  17. 2024Long Context RAG Performance of Large Language Modelscited by 23
  18. 20244D Panoptic Scene Graph Generationcited by 22
  19. 2024Measuring the Inconsistency of Large Language Models in Preferential Rankingcited by 19
  20. 2025ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelinescited by 16
  21. 2023ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Modelscited by 12
  22. 2023Emergent Causality and the Foundation of Consciousnesscited by 9
  23. 2023OpenCog Hyperon: A Framework for AGI at the Human Level and Beyondcited by 9
  24. 2021The General Theory of General Intelligence: A Pragmatic Patternist Perspectivecited by 8
  25. 2021Philosophical Specification of Empathetic Ethical Artificial Intelligencecited by 8
  26. 2023Computational Dualism and Objective Superintelligencecited by 8
  27. 2021Compression, The Fermi Paradox and Artificial Super-Intelligencecited by 7
  28. 2024What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generationcited by 6
  29. 2024Are Biological Systems More Intelligent Than Artificial Intelligence?cited by 5
  30. 2023Robustifying Language Models with Test-Time Adaptationcited by 2
  31. 2025Do Language Models Have Bayesian Brains? Distinguishing Stochastic and Deterministic Decision Patterns within Large Language Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanismscited by 4
  2. 2026Runtime-Independent Persistent Agents: Preserving Identity, Memory, and Code Across Models, Harnesses, and Serverscited by 2
  3. 2026Measuring What Persists: Conditioning Mechanisms and a Geometric Framework for AI Agent Identitycited by 0
  4. 2026The AI Accountability Ecosystem in the Era of Language Modelscited by 0
  5. 2026Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Time, Identity and Consciousness in Language Model Agents
  2. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
  3. 2026Silent Failure in LLM Agent Systems: The Entropy Principle and the Inevitable Disorder of Autonomous Agents
  4. 2023Evaluating Language-Model Agents on Realistic Autonomous Tasks
  5. 2026A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems
  6. 2025How Do LLMs Fail In Agentic Scenarios? A Qualitative Analysis of Success and Failure Scenarios of Various LLMs in Agentic Simulations
  7. 2024Examining Identity Drift in Conversations of LLM Agents
  8. 2026Layered Mutability: Continuity and Governance in Persistent Self-Modifying Agents