mlsanityData to 14 Oct 2026

KnowRL: Teaching Language Models to Know What They Know

arXiv 2510.11407 ↗First posted 13 Oct 2025cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Truly reliable AI requires more than simply scaling up knowledge; it demands the ability to know what it knows and when it does not. Yet recent research shows that even the best LLMs misjudge their own competence in more than one in five cases, making any response born of such internal uncertainty impossible to fully trust. Inspired by self-improvement reinforcement learning techniques that require minimal data, we present a simple but powerful framework KnowRL that strengthens a model's internal understanding of its own feasibility boundaries, enabling safer and more responsible behaviour. Our framework combines two components: (i) introspection, where the model generates and classifies tasks it judges feasible or infeasible, and (ii) consensus-based rewarding, where stability of self-knowledge assessment is reinforced through internal agreement. By using internally generated data, this design strengthens consistency in self-knowledge and entirely avoids costly external supervision. In experiments on LLaMA-3.1-8B and Qwen-2.5-7B, KnowRL steadily improved self-knowledge, validated by both intrinsic self-consistency and extrinsic benchmarking. With nothing more than a small seed set and no external supervision, our method drove gains as high as 28% in accuracy and 12% in F1, outperforming baselines in just a few iterations. Our framework essentially unlocks the untapped capacity of LLMs to self-improve their knowledge awareness, opening the door to reliable, more accountable AI and safer deployment in critical applications. Owing to its simplicity and independence from external effort, we encourage applying this reliability-enhancing process to all future models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

38 of 47 references.

References · 38

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  5. 2022Large Language Models Can Self-Improvecited by 664
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  7. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  8. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  9. 2023Do Large Language Models Know What They Don't Know?cited by 243
  10. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  11. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
  12. 2024Rule Based Rewards for Language Model Safetycited by 108
26 more
  1. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 72
  2. 2024Self-Taught Evaluatorscited by 72
  3. 2023Self-Knowledge Guided Retrieval Augmentation for Large Language Modelscited by 69
  4. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59
  5. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 51
  6. 2025Self-Questioning Language Modelscited by 50
  7. 2023Benchmarking and Improving Generator-Validator Consistency of Language Modelscited by 44
  8. 2024Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigationcited by 44
  9. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 39
  10. 2025A Survey of Direct Preference Optimizationcited by 36
  11. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  12. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 30
  13. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  14. 2025Learning to Reason for Factualitycited by 24
  15. 2024LLM-Generated Black-box Explanations Can Be Adversarially Helpfulcited by 21
  16. 2024Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answeringcited by 20
  17. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16
  18. 2025Position: The Most Expensive Part of an LLM should be its Training Datacited by 16
  19. 2024Self-Cognition in Large Language Models: An Exploratory Studycited by 15
  20. 2025LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratchcited by 15
  21. 2024Question-Analysis Prompting Improves LLM Performance in Reasoning Taskscited by 13
  22. 2018Playing 20 Question Game with Policy-Based Reinforcement Learningcited by 9
  23. 2024Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflectioncited by 8
  24. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignmentcited by 4
  25. 2024What External Knowledge is Preferred by LLMs? Characterizing and Exploring Chain of Evidence in Imperfect Context for Multi-Hop QAcited by 3
  26. 2025Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledgecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  2. 2026Do Coding Agents Understand Least-Privilege Authorization?cited by 6
  3. 2025Look It Up: Analysing Internal Web Search Capabilities of Modern LLMscited by 3
  4. 2026Future Confidence Distillation in Large Language Modelscited by 1
  5. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback
  2. 2026Self-CTRL: Self-Consistency Training with Reinforcement Learning
  3. 2026Capability Self-Assessment in Large Language Models
  4. 2026ICRL: Learning to Internalize Self-Critique with Reinforcement Learning
  5. 2025RLSR: Reinforcement Learning from Self Reward
  6. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
  7. 2026KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance
  8. 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedback