mlsanityData to 14 Oct 2026

Human-in-the-Loop through Chain-of-Thought

arXiv 2306.07932 ↗First posted 10 Jun 2023cs.CL cs.AIcited by 20

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

While the emergence of powerful language models along with Chain-of-thought prompting has made automation more and more omnipresent, it sometimes demonstrates its weakness in long-term or multi-step logical reasoning. For example, users don't always get desirable answers for complex mathematical problems without human involvement. Against this background, we present the Manual Correction System (MCS) -- a human-in-the-loop system enhanced by Chain-of-Thought prompting, which explores how manual correction of sub-logics in rationales can improve LLM's reasoning performance. Moving one step forward, considering a system with human-in-the-loop involves more than having humans improve performance but also controlling the cost. Therefore, we post a Cost-utility Analysis Model for Human-in-the-Loop systems (CAMLOP) based on classical economics theory to analyze, quantify and balance the utility and the corresponding cost. We conduct experiments of MCS and CAMLOP with twelve datasets. A significant advantage w.r.t cost and utility proves its superiority over strong baselines.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 38 references that are in the map. First posted on arXiv 10 Jun 2023.

Builds on

17 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Large Language Models are Zero-Shot Reasoners24 May
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  8. 2018Hierarchical Neural Story Generationcited by 1,193
  9. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  10. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  11. 2016Solving General Arithmetic Word Problemscited by 360
  12. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
2 more
  1. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 149
  2. 2018Learning to Write with Cooperative Discriminatorscited by 119

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 7; 2025: 5; 2026: 3.

20 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  3. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  4. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 110
  5. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  6. 2024Large Language Model-based Human-Agent Collaboration for Complex Task Solvingcited by 38
  7. 2025The Future of Continual Learning in the Era of Foundation Models: Three Key Directionscited by 17
  8. 2025LLM-based Automated Grading with Human-in-the-Loopcited by 13
  9. 2023An Evaluation of ChatGPT-4's Qualitative Spatial Reasoning Capabilities in RCC-8cited by 9
  10. 2025Advancing the Scientific Method with Large Language Models: From Hypothesis to Discoverycited by 7
  11. 2024Can Large Language Models Reason about the Region Connection Calculus?cited by 5
  12. 2025How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyondcited by 4
8 more
  1. 2024EDDA: A Encoder-Decoder Data Augmentation Framework for Zero-Shot Stance Detectioncited by 3
  2. 2023Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPTcited by 2
  3. 2024VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awarenesscited by 2
  4. 2024Agentic AI: The Era of Semantic Decodingcited by 1
  5. 2026QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculicited by 1
  6. 2024PerkwE_COQA: Enhanced Persian Conversational Question Answering by combining contextual keyword extraction with Large Language Modelscited by 0
  7. 2026An Interactive Paradigm for Deep Researchcited by 0
  8. 2026Deep Interaction: An Efficient Human-AI Interaction Method for Large Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning
  2. 2023Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinking
  3. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
  4. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future
  5. 2025Chain of Draft: Thinking Faster by Writing Less
  6. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data
  7. 2026Cognitive Loop of Thought: Reversible Hierarchical Markov Chain for Efficient Mathematical Reasoning
  8. 2024Markov Chain of Thought for Efficient Mathematical Reasoning