mlsanityData to 7 Oct 2026

Generative transformations and patterns in LLM-native approaches for software verification and falsification

arXiv 2404.09384 ↗First posted 14 Apr 2024cs.SE cs.AI cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

The emergence of prompting as the dominant paradigm for leveraging Large Language Models (LLMs) has led to a proliferation of LLM-native software, where application behavior arises from complex, stochastic data transformations. However, the engineering of such systems remains largely exploratory and ad-hoc, hampered by the absence of conceptual frameworks, ex-ante methodologies, design guidelines, and specialized benchmarks. We argue that a foundational step towards a more disciplined engineering practice is a systematic understanding of the core functional units--generative transformations--and their compositional patterns within LLM-native applications. Focusing on the rich domain of software verification and falsification, we conduct a secondary study of over 100 research proposals to address this gap. We first present a fine-grained taxonomy of generative transformations, abstracting prompt-based interactions into conceptual signatures. This taxonomy serves as a scaffolding to identify recurrent transformation relationship patterns--analogous to software design patterns--that characterize solution approaches in the literature. Our analysis not only validates the utility of the taxonomy but also surfaces strategic gaps and cross-dimensional relationships, offering a structured foundation for future research in modular and compositional LLM application design, benchmarking, and the development of reliable LLM-native systems.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 145 of its 289 references that are in the map. First posted on arXiv 14 Apr 2024.

Builds on

145 of 289 references.

References · 145

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
38 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  13. 2023Teaching Large Language Models to Self-Debugcited by 963
  14. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  15. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  16. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  17. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  18. 2023Cognitive Architectures for Language Agentscited by 422
  19. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  20. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  21. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  22. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  23. 2022CodeT: Code Generation with Generated Testscited by 369
  24. 2019DocRED: A Large-Scale Document-Level Relation Extraction Datasetcited by 239
  25. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 213
  26. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  27. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
  28. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 163
  29. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  30. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  31. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  32. 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 142
  33. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
  34. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  35. 2019Fine-tune BERT for Extractive Summarizationcited by 122
  36. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 119
  37. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  38. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118

50 most cited of 145 All 145 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.