# PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMs

arXiv: https://arxiv.org/abs/2509.20105 · HTML: https://mlsanity.com/paper/2509.20105/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 31 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [Explaining Answers with Entailment Trees](https://mlsanity.com/paper/2104.08661/) (2021)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [Deductive Verification of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2306.03872/) (2023)
- [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2305.06984/) (2023)
- [Evaluating Step-by-step Reasoning Traces: A Survey](https://mlsanity.com/paper/2502.12289/) (2025)
- [LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities](https://mlsanity.com/paper/2504.16078/) (2025)
- [QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation](https://mlsanity.com/paper/2406.00132/) (2024)
- [Label Supervised LLaMA Finetuning](https://mlsanity.com/paper/2310.01208/) (2023)
- [Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers](https://mlsanity.com/paper/2403.01061/) (2024)
- [C3AI: Crafting and Evaluating Constitutions for Constitutional AI](https://mlsanity.com/paper/2502.15861/) (2025)
- [Reinforcement Learning from Human Feedback: Whose Culture, Whose Values, Whose Perspectives?](https://mlsanity.com/paper/2407.17482/) (2024)
- [Quantum Large Language Models via Tensor Network Disentanglers](https://mlsanity.com/paper/2410.17397/) (2024)
- [Quantum-Enhanced LLM Efficient Fine Tuning](https://mlsanity.com/paper/2503.12790/) (2025)
- [From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions](https://mlsanity.com/paper/2504.18691/) (2025)
- [LLM-Generated Tips Rival Expert-Created Tips in Helping Students Answer Quantum-Computing Questions](https://mlsanity.com/paper/2407.17024/) (2024)
- [Quantum-inspired Techniques in Tensor Networks for Industrial Contexts](https://mlsanity.com/paper/2404.11277/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) often struggle with maintaining coherent multi-step reasoning traces, particularly in tasks that require a structured logical flow. This work introduces a quantum-inspired approach to address the challenge by incorporating a fidelity-based reward derived from Projected Entangled Pair States (PEPS) into Proximal Policy Optimization. Unlike prior approaches that use direct supervision or contrastive objectives, the proposed method guides learning through structural consistency, offering a novel approach to enforce global coherence in generated reasoning traces. The proposed framework is evaluated using multiple coherence-determining metrics on diverse datasets such as GSM8K, StrategyQA, and EntailmentBank spanning arithmetic, intuitive, and entailment-based reasoning. Results show that the proposed quantum-inspired approach offers significant improvements over supervised, contrastive, and pretrained baseline approaches, highlighting the effectiveness of quantum-inspired fidelity as a foundation to improve reasoning trace coherence in LLMs.