# STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond

arXiv: https://arxiv.org/abs/2409.05367 · HTML: https://mlsanity.com/paper/2409.05367/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 105 references that are in the map.
- First posted on arXiv 9 Sep 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 15 of 15)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [Language Model Cascades](https://mlsanity.com/paper/2207.10342/) (2022)
- [MARG: Multi-Agent Review Generation for Scientific Papers](https://mlsanity.com/paper/2401.04259/) (2024)
- [Demystifying Chains, Trees, and Graphs of Thoughts](https://mlsanity.com/paper/2401.14295/) (2024)
- [AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews](https://mlsanity.com/paper/2408.10365/) (2024)
- [LLM-as-a-tutor in EFL Writing Education: Focusing on Evaluation of Student-LLM Interaction](https://mlsanity.com/paper/2310.05191/) (2023)
- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models](https://mlsanity.com/paper/2407.13934/) (2024)
- [Large Language Model Programs](https://mlsanity.com/paper/2305.05364/) (2023)
- [DREsS: Dataset for Rubric-based Essay Scoring on EFL Writing](https://mlsanity.com/paper/2402.16733/) (2024)
- [An Explainable AI Approach to Large Language Model Assisted Causal Model Auditing and Development](https://mlsanity.com/paper/2312.16211/) (2023)
- [MultiSChuBERT: Effective Multimodal Fusion for Scholarly Document Quality Prediction](https://mlsanity.com/paper/2308.07971/) (2023)
- [Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective](https://mlsanity.com/paper/2109.01190/) (2021)

## Built on by (most cited first, 5 of 5)

- [Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback](https://mlsanity.com/paper/2508.10795/) (2025)
- [Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework](https://mlsanity.com/paper/2508.21422/) (2025)
- [Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future](https://mlsanity.com/paper/2604.27924/) (2026)
- [Preventing the Collapse of Peer Review Requires Verification-First AI](https://mlsanity.com/paper/2601.16909/) (2026)
- [From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation](https://mlsanity.com/paper/2603.17588/) (2026)

## Abstract (arXiv)

Critical text assessment is at the core of many expert activities, such as fact-checking, peer review, and essay grading. Yet, existing work treats critical text assessment as a black box problem, limiting interpretability and human-AI collaboration. To close this gap, we introduce Structured Reasoning In Critical Text Assessment (STRICTA), a novel specification framework to model text assessment as an explicit, step-wise reasoning process. STRICTA breaks down the assessment into a graph of interconnected reasoning steps drawing on causality theory (Pearl, 1995). This graph is populated based on expert interaction data and used to study the assessment process and facilitate human-AI collaboration. We formally define STRICTA and apply it in a study on biomedical paper assessment, resulting in a dataset of over 4000 reasoning steps from roughly 40 biomedical experts on more than 20 papers. We use this dataset to empirically study expert reasoning in critical text assessment, and investigate if LLMs are able to imitate and support experts within these workflows. The resulting tools and datasets pave the way for studying collaborative expert-AI reasoning in text assessment, in peer review and beyond.