Generative transformations and patterns in LLM-native approaches for software verification and falsification
Not on the map. Why?
Abstract (arXiv)
The emergence of prompting as the dominant paradigm for leveraging Large Language Models (LLMs) has led to a proliferation of LLM-native software, where application behavior arises from complex, stochastic data transformations. However, the engineering of such systems remains largely exploratory and ad-hoc, hampered by the absence of conceptual frameworks, ex-ante methodologies, design guidelines, and specialized benchmarks. We argue that a foundational step towards a more disciplined engineering practice is a systematic understanding of the core functional units--generative transformations--and their compositional patterns within LLM-native applications. Focusing on the rich domain of software verification and falsification, we conduct a secondary study of over 100 research proposals to address this gap. We first present a fine-grained taxonomy of generative transformations, abstracting prompt-based interactions into conceptual signatures. This taxonomy serves as a scaffolding to identify recurrent transformation relationship patterns--analogous to software design patterns--that characterize solution approaches in the literature. Our analysis not only validates the utility of the taxonomy but also surfaces strategic gaps and cross-dimensional relationships, offering a structured foundation for future research in modular and compositional LLM application design, benchmarking, and the development of reliable LLM-native systems.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 145 of its 289 references that are in the map. First posted on arXiv 14 Apr 2024.
Builds on
145 of 289 references.
References · 145
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
38 more
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2019The Curious Case of Neural Text Degenerationcited by 2,881
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2023A Survey on Evaluation of Large Language Modelscited by 1,227
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
- 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
- 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
- 2023Teaching Large Language Models to Self-Debugcited by 963
- 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
- 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
- 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
- 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
- 2023Cognitive Architectures for Language Agentscited by 422
- 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
- 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
- 2023Learning to Compress Prompts with Gist Tokenscited by 379
- 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
- 2022CodeT: Code Generation with Generated Testscited by 369
- 2019DocRED: A Large-Scale Document-Level Relation Extraction Datasetcited by 239
- 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 213
- 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
- 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
- 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 163
- 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
- 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
- 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
- 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 142
- 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
- 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
- 2019Fine-tune BERT for Extractive Summarizationcited by 122
- 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 119
- 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
- 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
50 most cited of 145 All 145 (JSON)
Built on by
Citing papers per year: 2025: 2.
2 papers.