# Integrating Large Language Models and Reinforcement Learning for Non-Linear Reasoning

arXiv: https://arxiv.org/abs/2410.13501 · HTML: https://mlsanity.com/paper/2410.13501/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [Large Language Model Guided Tree-of-Thought](https://mlsanity.com/paper/2305.08291/) (2023)
- [MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning](https://mlsanity.com/paper/2205.00445/) (2022)
- [Learning to Generate Better Than Your LLM](https://mlsanity.com/paper/2306.11816/) (2023)
- [CodeMind: Evaluating Large Language Models for Code Reasoning](https://mlsanity.com/paper/2402.09664/) (2024)
- [Towards Understanding the Capability of Large Language Models on Code Clone Detection: A Survey](https://mlsanity.com/paper/2308.01191/) (2023)
- [A systematic literature review on source code similarity measurement and clone detection: techniques, applications, and challenges](https://mlsanity.com/paper/2306.16171/) (2023)
- [Large Language Model Programs](https://mlsanity.com/paper/2305.05364/) (2023)
- [Evaluating the Effectiveness of GPT-4 Turbo in Creating Defeaters for Assurance Cases](https://mlsanity.com/paper/2401.17991/) (2024)
- [An empirical study of ChatGPT-3.5 on question answering and code maintenance](https://mlsanity.com/paper/2310.02104/) (2023)
- [Equivalence of Dataflow Graphs via Rewrite Rules Using a Graph-to-Sequence Neural Model](https://mlsanity.com/paper/2002.06799/) (2020)
- [Proving Equivalence Between Complex Expressions Using Graph-to-Sequence Neural Models](https://mlsanity.com/paper/2106.02452/) (2021)

## Built on by (most cited first, 1 of 1)

- [CodePivot: Bootstrapping Multilingual Transpilation in LLMs via Reinforcement Learning without Parallel Corpora](https://mlsanity.com/paper/2604.18027/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) were shown to struggle with long-term planning, which may be caused by the limited way in which they explore the space of possible solutions. We propose an architecture where a Reinforcement Learning (RL) Agent guides an LLM's space exploration: (1) the Agent has access to domain-specific information, and can therefore make decisions about the quality of candidate solutions based on specific and relevant metrics, which were not explicitly considered by the LLM's training objective; (2) the LLM can focus on generating immediate next steps, without the need for long-term planning. We allow non-linear reasoning by exploring alternative paths and backtracking. We evaluate this architecture on the program equivalence task, and compare it against Chain of Thought (CoT) and Tree of Thoughts (ToT). We assess both the downstream task, denoting the binary classification, and the intermediate reasoning steps. Our approach compares positively against CoT and ToT.