# Balancing Exploration and Exploitation in LLM using Soft RLLF for Enhanced Negation Understanding

arXiv: https://arxiv.org/abs/2403.01185 · HTML: https://mlsanity.com/paper/2403.01185/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 2 Mar 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [Understanding by Understanding Not: Modeling Negation in Language Models](https://mlsanity.com/paper/2105.03519/) (2021)
- [Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence](https://mlsanity.com/paper/2306.07075/) (2023)
- [An Analysis of Negation in Natural Language Understanding Corpora](https://mlsanity.com/paper/2203.08929/) (2022)
- [CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation](https://mlsanity.com/paper/2211.00295/) (2022)
- [Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Models](https://mlsanity.com/paper/2307.11224/) (2023)
- [A negation detection assessment of GPTs: analysis with the xNot360 dataset](https://mlsanity.com/paper/2306.16638/) (2023)
- [Enhancing Logical Reasoning in Large Language Models to Facilitate Legal Applications](https://mlsanity.com/paper/2311.13095/) (2023)

## Built on by (most cited first, 3 of 3)

- [Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering](https://mlsanity.com/paper/2411.11504/) (2024)
- [Adversarial Testing in LLMs: Insights into Decision-Making Vulnerabilities](https://mlsanity.com/paper/2505.13195/) (2025)
- [An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems](https://mlsanity.com/paper/2501.00562/) (2024)

## Abstract (arXiv)

Finetuning approaches in NLP often focus on exploitation rather than exploration, which may lead to suboptimal models. Given the vast search space of natural language, this limited exploration can restrict their performance in complex, high-stakes domains, where accurate negation understanding and logical reasoning abilities are crucial. To address this issue, we leverage Reinforcement Learning from Logical Feedback (RLLF) to create an effective balance between exploration and exploitation in LLMs. Our approach employs an appropriate benchmark dataset for training and evaluation, highlighting the importance of exploration in enhancing negation understanding capabilities. We compare the performance of our RLLF-enhanced LLMs with baseline models trained without RLLF, demonstrating the value of this balanced approach. Furthermore, we showcase the potential of our method in legal AI applications by employing transfer learning and evaluating its impact on negation understanding. Our experimental results exhibit the effectiveness of balancing exploration and exploitation with RLLF in improving LLMs' negation capabilities. This has implications for the development of more accurate, reliable, and logically consistent language models in high-stakes domains.