# KnowRL: Teaching Language Models to Know What They Know

arXiv: https://arxiv.org/abs/2510.11407 · HTML: https://mlsanity.com/paper/2510.11407/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 47 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 30 of 38)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints](https://mlsanity.com/paper/2309.16240/) (2023)
- [Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation](https://mlsanity.com/paper/2307.11019/) (2023)
- [Rule Based Rewards for Language Model Safety](https://mlsanity.com/paper/2411.01111/) (2024)
- [Asymmetric self-play for automatic goal discovery in robotic manipulation](https://mlsanity.com/paper/2101.04882/) (2021)
- [Self-Taught Evaluators](https://mlsanity.com/paper/2408.02666/) (2024)
- [Self-Knowledge Guided Retrieval Augmentation for Large Language Models](https://mlsanity.com/paper/2310.05002/) (2023)
- [When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation](https://mlsanity.com/paper/2402.11457/) (2024)
- [SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data](https://mlsanity.com/paper/2505.20347/) (2025)
- [Self-Questioning Language Models](https://mlsanity.com/paper/2508.03682/) (2025)
- [Benchmarking and Improving Generator-Validator Consistency of Language Models](https://mlsanity.com/paper/2310.01846/) (2023)
- [Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation](https://mlsanity.com/paper/2401.15449/) (2024)
- [Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety](https://mlsanity.com/paper/2503.05021/) (2025)
- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training](https://mlsanity.com/paper/2505.22453/) (2025)
- [Language Models Fail to Introspect About Their Knowledge of Language](https://mlsanity.com/paper/2503.07513/) (2025)
- [Undesirable Memorization in Large Language Models: A Survey](https://mlsanity.com/paper/2410.02650/) (2024)
- [Learning to Reason for Factuality](https://mlsanity.com/paper/2508.05618/) (2025)
- [LLM-Generated Black-box Explanations Can Be Adversarially Helpful](https://mlsanity.com/paper/2405.06800/) (2024)
- [Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering](https://mlsanity.com/paper/2405.14383/) (2024)
- [Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models](https://mlsanity.com/paper/2406.04274/) (2024)
- [Position: The Most Expensive Part of an LLM should be its Training Data](https://mlsanity.com/paper/2504.12427/) (2025)

## Built on by (most cited first, 5 of 5)

- [Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skills](https://mlsanity.com/paper/2512.16301/) (2025)
- [Do Coding Agents Understand Least-Privilege Authorization?](https://mlsanity.com/paper/2605.14859/) (2026)
- [Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs](https://mlsanity.com/paper/2511.18931/) (2025)
- [Future Confidence Distillation in Large Language Models](https://mlsanity.com/paper/2607.07626/) (2026)
- [A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions](https://mlsanity.com/paper/2604.17312/) (2026)

## Abstract (arXiv)

Truly reliable AI requires more than simply scaling up knowledge; it demands the ability to know what it knows and when it does not. Yet recent research shows that even the best LLMs misjudge their own competence in more than one in five cases, making any response born of such internal uncertainty impossible to fully trust. Inspired by self-improvement reinforcement learning techniques that require minimal data, we present a simple but powerful framework KnowRL that strengthens a model's internal understanding of its own feasibility boundaries, enabling safer and more responsible behaviour. Our framework combines two components: (i) introspection, where the model generates and classifies tasks it judges feasible or infeasible, and (ii) consensus-based rewarding, where stability of self-knowledge assessment is reinforced through internal agreement. By using internally generated data, this design strengthens consistency in self-knowledge and entirely avoids costly external supervision. In experiments on LLaMA-3.1-8B and Qwen-2.5-7B, KnowRL steadily improved self-knowledge, validated by both intrinsic self-consistency and extrinsic benchmarking. With nothing more than a small seed set and no external supervision, our method drove gains as high as 28% in accuracy and 12% in F1, outperforming baselines in just a few iterations. Our framework essentially unlocks the untapped capacity of LLMs to self-improve their knowledge awareness, opening the door to reliable, more accountable AI and safer deployment in critical applications. Owing to its simplicity and independence from external effort, we encourage applying this reliability-enhancing process to all future models.