mlsanityData to 14 Oct 2026

DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis

arXiv 2609.11504 ↗First posted 10 Sep 2026cs.LG cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

A structurally valid DeFi workflow can still authorize a costly trade. We introduce DeFiFlowBench, a benchmark of 207 team-authored prompts for natural-language DeFi workflow synthesis. It measures graph coverage, configuration completeness, and declared safety predicates, then tests supported trade configurations on a local EVM. Direct, constrained, and few-shot prompting produce 14-19 unsafe held-out executions per configuration under a fixed 5% price-impact cap. A slippage bound derived from a quote does not prevent the price impact of the order itself. We propose Koan-Safe, which combines a prompt-only intent parser, a replaceable generator, and structural repair with default safety parameters. On 75 held-out workflow prompts, its hybrid variant scores 0.67 on the static safety proxy, compared with 0.33 for the best baseline. Koan-Safe records no unsafe executions on the saved benchmark outputs. A matched-candidate ablation produces 14-17 unsafe executions when enforcement is disabled. Additional tests expose the limits of default injection: permissive existing thresholds can still authorize unsafe trades. A separately evaluated policy cap addresses this failure on a 36-case diagnostic grid. These results support explicit trade protections and execution-based evaluation, while distinguishing declared safety from a general guarantee.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

23 of 31 references.

References · 23

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  4. 2018Datasheets for Datasetscited by 1,386
  5. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  6. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  7. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  8. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  9. 2024AFlow: Automating Agentic Workflow Generationcited by 432
  10. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 306
  11. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  12. 2023An LLM Compiler for Parallel Function Callingcited by 179
11 more
  1. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  2. 2015Shield Synthesis: Runtime Enforcement for Reactive Systemscited by 42
  3. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agentscited by 40
  4. 2024Benchmarking Agentic Workflow Generationcited by 37
  5. 2024FlowMind: Automatic Workflow Generation with LLMscited by 30
  6. 2021SoK: Decentralized Finance (DeFi)cited by 23
  7. 2026Agentproof: Static Verification of Agent Workflow Graphscited by 9
  8. 2021DeFiRanger: Detecting Price Manipulation Attacks on DeFi Applicationscited by 5
  9. 2026Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Languagecited by 3
  10. 2026Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactionscited by 1
  11. 2026PACE: Policy-Attested Contract Execution for Safe AI Agents in Decentralized Financecited by 1

Built on by

Not cited yet.