mlsanityData to 14 Oct 2026

EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation

arXiv 2601.06565 ↗First posted 10 Jan 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models are increasingly applied to various development scenarios. However, in on-chain transaction scenarios, even a minor error can cause irreversible loss for users. Existing evaluations often overlook execution accuracy and safety. We introduce EVM-QuestBench, an execution-grounded benchmark for natural-language transaction-script generation on EVM-compatible chains. The benchmark employs dynamic evaluation: instructions are sampled from template pools, numeric parameters are drawn from predefined intervals, and validators verify outcomes against these instantiated values. EVM-QuestBench contains 107 tasks (62 atomic, 45 composite). Its modular architecture enables rapid task development. The runner executes scripts on a forked EVM chain with snapshot isolation; composite tasks apply step-efficiency decay. We evaluate 20 models and find large performance gaps, with split scores revealing persistent asymmetry between single-action precision and multi-step workflow completion. Code: https://anonymous.4open.science/r/bsc_quest_bench-A9CF/.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 14 of its 20 references that are in the map. First posted on arXiv 10 Jan 2026.

Builds on

14 of 20 references.

References · 14

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2023PaLM 2 Technical Reportcited by 1,162
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  11. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  12. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 91
2 more
  1. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 47
  2. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMscited by 4