# To Reason or Not to: Selective Chain-of-Thought in Medical Question Answering

arXiv: https://arxiv.org/abs/2602.20130 · HTML: https://mlsanity.com/paper/2602.20130/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 8 of its 31 references that are in the map.
- First posted on arXiv 23 Feb 2026.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 8 of 8)

- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Automating Expert-Level Medical Reasoning Evaluation of Large Language Models](https://mlsanity.com/paper/2507.07988/) (2025)
- [Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications](https://mlsanity.com/paper/2508.00669/) (2025)
- [An evaluation of DeepSeek Models in Biomedical Natural Language Processing](https://mlsanity.com/paper/2503.00624/) (2025)
- [Instruction Tuning and CoT Prompting for Contextual Medical QA with LLMs](https://mlsanity.com/paper/2506.12182/) (2025)
- [MedCoT-RAG: Causal Chain-of-Thought RAG for Medical Question Answering](https://mlsanity.com/paper/2508.15849/) (2025)
- [Benchmarking GPT-5 for biomedical natural language processing](https://mlsanity.com/paper/2509.04462/) (2025)

## Abstract (arXiv)

Objective: To improve the efficiency of medical question answering (MedQA) with large language models (LLMs) by avoiding unnecessary reasoning while maintaining accuracy. Methods: We propose Selective Chain-of-Thought (Selective CoT), an inference-time strategy that first predicts whether a question requires reasoning and generates a rationale only when needed. Two open-source LLMs (Llama-3.1-8B and Qwen-2.5-7B) were evaluated on four biomedical QA benchmarks-HeadQA, MedQA-USMLE, MedMCQA, and PubMedQA. Metrics included accuracy, total generated tokens, and inference time. Results: Selective CoT reduced inference time by 13-45% and token usage by 8-47% with minimal accuracy loss (≤4%). In some model-task pairs, it achieved both higher accuracy and greater efficiency than standard CoT. Compared with fixed-length CoT, Selective CoT reached similar or superior accuracy at substantially lower computational cost. Discussion: Selective CoT dynamically balances reasoning depth and efficiency by invoking explicit reasoning only when beneficial, reducing redundancy on recall-type questions while preserving interpretability. Conclusion: Selective CoT provides a simple, model-agnostic, and cost-effective approach for medical QA, aligning reasoning effort with question complexity to enhance real-world deployability of LLM-based clinical systems.