Quality-Driven Agentic Reasoning for LLM-Assisted Software Design: Questions-of-Thoughts (QoT) as a Time-Series Self-QA Chain
On the map: LLM reasoning → Reasoning for agents, planning and code
also in 1 more
Abstract (arXiv)
Recent advances in large language models (LLMs) have accelerated AI-assisted software development, yet practical deployment remains constrained by incomplete implementations, weak modularization, and inconsistent security practices. We introduce Questions-of-Thoughts (QoT), a quality-driven inference-time scaffold that turns a user goal into (i) an ordered sequence of engineering steps and (ii) stepwise self-questioning to verify constraints and reduce omission errors, while maintaining a lightweight reasoning record that stabilizes subsequent design decisions. We evaluate QoT across three representative backend engineering domains: API Design, Data Communication, and File Systems. Each task requires multi-module decomposition and exposes standard failure modes in LLM-generated systems. To enable data-driven comparison, we score generated artifacts using an ISO/IEC-inspired quality rubric that measures Scalability, Completeness, Modularity, and Security. We report domain-wise gains as the change in total quality score, defined as the QoT score minus the NoQoT score. Results show capacity-dependent improvements: QoT yields consistent quality improvements for larger models and more complex domains, while smaller models may exhibit trade-offs under tight context and planning budgets. We release an open artifact with prompts, scoring guidelines, raw generations, and scripts that reproduce the reported tables and figures to support applied AI and data analytics research.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 15 of its 17 references that are in the map. First posted on arXiv 10 Mar 2026.
Builds on
15 of 17 references.
Closest ancestors · 2
Same direction, newest first
- 2025S*: Test Time Scaling for Code Generation20 Feb
- 2024CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models7 Nov
Other references · 13
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
- 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
- 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
- 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
- 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 69
- 2024You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projectscited by 45
- 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 24
- 2024Divide-Verify-Refine: Can LLMs Self-Align with Complex Instructions?cited by 7
1 more
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents
- 2025Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents
- 2025Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol
- 2025QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks
- 2025Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationale
- 2026Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment
- 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System
- 2025Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories