# Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactions

arXiv: https://arxiv.org/abs/2604.27763 · HTML: https://mlsanity.com/paper/2604.27763/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 14 of 14)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models](https://mlsanity.com/paper/2310.08659/) (2023)
- [On the Tool Manipulation Capability of Open-source Large Language Models](https://mlsanity.com/paper/2305.16504/) (2023)
- [HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation](https://mlsanity.com/paper/2412.21199/) (2024)
- [SoK: Decentralized Exchanges (DEX) with Automated Market Maker (AMM) Protocols](https://mlsanity.com/paper/2103.12732/) (2021)
- [Modeling and Understanding Ethereum Transaction Records via a Complex Network Approach](https://mlsanity.com/paper/2012.15462/) (2020)
- [Characterizing Code Clones in the Ethereum Smart Contract Ecosystem](https://mlsanity.com/paper/1905.00272/) (2019)
- [API Pack: A Massive Multi-Programming Language Dataset for API Call Generation](https://mlsanity.com/paper/2402.09615/) (2024)
- [Disentangling Decentralized Finance (DeFi) Compositions](https://mlsanity.com/paper/2111.11933/) (2021)
- [Is my transaction done yet? An empirical study of transaction processing times in the Ethereum Blockchain Platform](https://mlsanity.com/paper/2206.08959/) (2022)

## Built on by (most cited first, 1 of 1)

- [DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis](https://mlsanity.com/paper/2609.11504/) (2026)

## Abstract (arXiv)

The emergence of Large Language Models (LLMs) offers a transformative interface for Web3, yet existing benchmarks fail to capture the complexity of translating high-level user intents into functionally correct, state-dependent on-chain transactions. We present textscIntent2Tx, a high-fidelity benchmark featuring 29,921 single-step and 1,575 multi-step instances meticulously derived from 300 days of real-world Ethereum mainnet traces. Unlike prior works that rely on synthetic instructions, textscIntent2Tx grounds natural language intents in real-world protocol interactions across 11 categories, including diverse long-tail Decentralized Finance (DeFi) primitives. To enable rigorous evaluation, we propose an execution-aware framework that transcends surface-level text matching by employing differential state analysis on forked mainnet environments. Our extensive evaluation of 16 state-of-the-art LLMs reveals that while scaling and retrieval-augmentation enhance logical consistency and parameter precision, current models struggle with out-of-distribution generalization and multi-step planning. Crucially, our execution-based analysis demonstrates that syntactically valid outputs often fail to achieve intended state transitions, highlighting a significant gap in current "reasoning-to-execution" capabilities. textscIntent2Tx serves as a critical foundation for developing autonomous, reliable agents in intent-centric Web3 ecosystems. Code and data: https://anonymous.4open.science/r/Intent2TxBench-97FF .