mlsanityData to 7 Oct 2026

DRIP-R: A Benchmark for Decision-Making and Reasoning Under Real-World Policy Ambiguity in the Retail Domain

arXiv 2605.07699 ↗First posted 8 May 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

LLM-based agents are increasingly deployed for routine but consequential tasks in real-world domains, where their behavior is governed by inherently ambiguous domain policies that admit multiple valid interpretations. Despite the prevalence of such ambiguities in practice, existing agent benchmarks largely assume unambiguous, well-specified policies, leaving a critical evaluation gap. We introduce DRIP-R, a benchmark that systematically exploits real-world retail policy ambiguities to construct scenarios in which no single correct resolution exists. DRIP-R comprises a curated set of policy-ambiguous return scenarios paired with a realistic customer personas, a full-duplex conversational simulation with tool-calling capabilities and a multi-judge evaluation framework covering policy adherence, dialogue quality, behavioral alignment, and resolution quality. Our experiments show that frontier models fundamentally disagree on identical policy-ambiguous scenarios, confirming that ambiguity poses a genuine and systematic challenge to LLM decision-making.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 21 of its 59 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

21 of 59 references.

References · 21

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2024A Survey on LLM-as-a-Judgecited by 1,384
  4. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  5. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  6. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  7. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  8. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  9. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  10. 2025Survey on Evaluation of LLM-based Agentscited by 190
  11. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  12. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 31
9 more
  1. 2026τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domainscited by 30
  2. 2023Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Gamescited by 24
  3. 2025Towards Enforcing Company Policy Adherence in Agentic Workflowscited by 17
  4. 2026Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intentscited by 11
  5. 2024AmbigNLG: Addressing Task Ambiguity in Instruction for NLGcited by 10
  6. 2025MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactionscited by 7
  7. 2026AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wildcited by 7
  8. 2024Making Task-Oriented Dialogue Datasets More Natural by Synthetically Generating Indirect User Requestscited by 2
  9. 2026GAIN: A Benchmark for Goal-Aligned Decision-Making of Large Language Models under Imperfect Normscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments
  2. 2026RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments
  3. 2026Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomes
  4. 2026PA3: Policy-Aware Agent Alignment through Chain-of-Thought
  5. 2026PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents
  6. 2026On Evaluating and Improving Conversational Agents in Production
  7. 2025AIM-Bench: Evaluating Decision-making Biases of Agentic LLM as Inventory Manager
  8. 2026LLM Consumer Behavior Theory: Foundations of a Novel Research Field