mlsanityData to 7 Oct 2026

BLAST: Benchmarking LLMs with ASP-based Structured Testing

arXiv 2604.22306 ↗First posted 24 Apr 2026cs.LO cs.AI cs.PLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable performance across a broad spectrum of tasks, including natural language understanding, dialogue systems, and code generation. Despite evident progress, less attention has been paid to their effectiveness in handling declarative paradigms such as Answer Set Programming (ASP), to date. In this paper we introduce BLAST: The first dedicated benchmarking methodology and associated dataset for evaluating the accuracy of LLMs in generating ASP code. BLAST provides a structured evaluation framework featuring two novel semantic metrics tailored to ASP code generation. The paper presents the results of an empirical evaluation involving ten well-established graph-related problems from the ASP literature and a diverse set of eight state-of-the-art LLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 55 references that are in the map. First posted on arXiv 24 Apr 2026.

Builds on

27 of 55 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2024A Survey on Large Language Models for Code Generationcited by 646
  6. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  7. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 264
  8. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  9. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  10. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
  11. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  12. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworkscited by 36
15 more
  1. 2023DialogBench: Evaluating LLMs as Human-like Dialogue Systemscited by 31
  2. 2020The ILASP system for Inductive Learning of Answer Set Programscited by 27
  3. 2025Ontology Generation using Large Language Modelscited by 23
  4. 2015A Survey and Classification of Controlled Natural Languagescited by 18
  5. 2022AI-driven Development Is Here: Should You Worry?cited by 15
  6. 2024StackEval: Benchmarking LLMs in Coding Assistancecited by 14
  7. 2019The Seventh Answer Set Programming Competition: Design and Resultscited by 11
  8. 2024Towards Automatic Composition of ASP Programs from Natural Language Specificationscited by 7
  9. 2023CNL2ASP: converting controlled natural language sentences into ASPcited by 5
  10. 2018Specifying and Verbalising Answer Set Programs in Controlled Natural Languagecited by 4
  11. 2023Clingraph: A System for ASP-based Visualizationcited by 4
  12. 2023Can GPT models Follow Human Summarization Guidelines? A Study for Targeted Communication Goalscited by 4
  13. 2025A Solver-in-the-Loop Framework for Improving LLMs on Answer Set Programming for Logic Puzzle Solvingcited by 4
  14. 2024Unit Testing in ASP Revisited: Language and Test-Driven Development Environmentcited by 2
  15. 2025Visual Graph Question Answering with ASP and LLMs for Language Parsingcited by 2

Built on by

Not cited yet.