mlsanityData to 7 Oct 2026

Quality-Driven Agentic Reasoning for LLM-Assisted Software Design: Questions-of-Thoughts (QoT) as a Time-Series Self-QA Chain

arXiv 2603.11082 ↗First posted 10 Mar 2026cs.SE cs.AIcited by 1

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Recent advances in large language models (LLMs) have accelerated AI-assisted software development, yet practical deployment remains constrained by incomplete implementations, weak modularization, and inconsistent security practices. We introduce Questions-of-Thoughts (QoT), a quality-driven inference-time scaffold that turns a user goal into (i) an ordered sequence of engineering steps and (ii) stepwise self-questioning to verify constraints and reduce omission errors, while maintaining a lightweight reasoning record that stabilizes subsequent design decisions. We evaluate QoT across three representative backend engineering domains: API Design, Data Communication, and File Systems. Each task requires multi-module decomposition and exposes standard failure modes in LLM-generated systems. To enable data-driven comparison, we score generated artifacts using an ISO/IEC-inspired quality rubric that measures Scalability, Completeness, Modularity, and Security. We report domain-wise gains as the change in total quality score, defined as the QoT score minus the NoQoT score. Results show capacity-dependent improvements: QoT yields consistent quality improvements for larger models and more complex domains, while smaller models may exhibit trade-offs under tight context and planning budgets. We release an open artifact with prompts, scoring guidelines, raw generations, and scripts that reproduce the reported tables and figures to support applied AI and data analytics research.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 15 of its 17 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

15 of 17 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025S*: Test Time Scaling for Code Generation20 Feb
  2. 2024CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models7 Nov

Other references · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  6. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
  7. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  8. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  9. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 69
  10. 2024You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projectscited by 45
  11. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 24
  12. 2024Divide-Verify-Refine: Can LLMs Self-Align with Complex Instructions?cited by 7
1 more
  1. 2025CodeGrad: Integrating Multi-Step Verification with Gradient-Based LLM Refinementcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents
  2. 2025Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents
  3. 2025Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol
  4. 2025QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks
  5. 2025Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationale
  6. 2026Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment
  7. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System
  8. 2025Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories