# Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning

arXiv: https://arxiv.org/abs/2312.17484 · HTML: https://mlsanity.com/paper/2312.17484/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 29 Dec 2023.

## Builds on (most cited first, 11 of 11)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [More Control for Free! Image Synthesis with Semantic Diffusion Guidance](https://mlsanity.com/paper/2112.05744/) (2021)
- [Learning to Generate Reviews and Discovering Sentiment](https://mlsanity.com/paper/1704.01444/) (2017)
- [Language Models are Open Knowledge Graphs](https://mlsanity.com/paper/2010.11967/) (2020)

## Built on by (most cited first, 30 of 45)

- [Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps](https://mlsanity.com/paper/2407.07071/) (2024)
- [TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space](https://mlsanity.com/paper/2402.17811/) (2024)
- [Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors](https://mlsanity.com/paper/2410.12299/) (2024)
- [Spectral Editing of Activations for Large Language Model Alignment](https://mlsanity.com/paper/2405.09719/) (2024)
- [Internal Consistency and Self-Feedback in Large Language Models: A Survey](https://mlsanity.com/paper/2407.14507/) (2024)
- [Steer LLM Latents for Hallucination Detection](https://mlsanity.com/paper/2503.01917/) (2025)
- [Knowledge Boundary of Large Language Models: A Survey](https://mlsanity.com/paper/2412.12472/) (2024)
- [Personalized Text Generation with Contrastive Activation Steering](https://mlsanity.com/paper/2503.05213/) (2025)
- [PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration](https://mlsanity.com/paper/2406.01394/) (2024)
- [HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild](https://mlsanity.com/paper/2403.04307/) (2024)
- [Mitigating Large Language Model Hallucination with Faithful Finetuning](https://mlsanity.com/paper/2406.11267/) (2024)
- [On the Universal Truthfulness Hyperplane Inside LLMs](https://mlsanity.com/paper/2407.08582/) (2024)
- [Spherical Steering: Geometry-Aware Activation Rotation for Language Models](https://mlsanity.com/paper/2602.08169/) (2026)
- [Constructing Benchmarks and Interventions for Combating Hallucinations in LLMs](https://mlsanity.com/paper/2404.09971/) (2024)
- [Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparators](https://mlsanity.com/paper/2408.12325/) (2024)
- [Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness](https://mlsanity.com/paper/2404.00216/) (2024)
- [Distinguishing Ignorance from Error in LLM Hallucinations](https://mlsanity.com/paper/2410.22071/) (2024)
- [Non-Linear Inference Time Intervention: Improving LLM Truthfulness](https://mlsanity.com/paper/2403.18680/) (2024)
- [NoVo: Norm Voting off Hallucinations with Attention Heads in Large Language Models](https://mlsanity.com/paper/2410.08970/) (2024)
- [LF-Steering: Latent Feature Activation Steering for Enhancing Semantic Consistency in Large Language Models](https://mlsanity.com/paper/2501.11036/) (2025)
- [SAFE: A Sparse Autoencoder-Based Framework for Robust Query Enrichment and Hallucination Mitigation in LLMs](https://mlsanity.com/paper/2503.03032/) (2025)
- [Steering When Necessary: Flexible Steering Large Language Models with Backtracking](https://mlsanity.com/paper/2508.17621/) (2025)
- [TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Models](https://mlsanity.com/paper/2510.01274/) (2025)
- [Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models](https://mlsanity.com/paper/2507.02870/) (2025)
- [GRATH: Gradual Self-Truthifying for Large Language Models](https://mlsanity.com/paper/2401.12292/) (2024)
- [Enhanced Language Model Truthfulness with Learnable Intervention and Uncertainty Expression](https://mlsanity.com/paper/2405.00301/) (2024)
- [Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions](https://mlsanity.com/paper/2509.23782/) (2025)
- [Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis](https://mlsanity.com/paper/2412.02946/) (2024)
- [ExpertSteer: Intervening in LLMs through Expert Knowledge](https://mlsanity.com/paper/2505.12313/) (2025)
- [DSVD: Dynamic Self-Verify Decoding for Faithful Generation in Large Language Models](https://mlsanity.com/paper/2503.03149/) (2025)

## Abstract (arXiv)

Despite the great success of large language models (LLMs) in various tasks, they suffer from generating hallucinations. We introduce Truth Forest, a method that enhances truthfulness in LLMs by uncovering hidden truth representations using multi-dimensional orthogonal probes. Specifically, it creates multiple orthogonal bases for modeling truth by incorporating orthogonal constraints into the probes. Moreover, we introduce Random Peek, a systematic technique considering an extended range of positions within the sequence, reducing the gap between discerning and generating truth features in LLMs. By employing this approach, we improved the truthfulness of Llama-2-7B from 40.8% to 74.5% on TruthfulQA. Likewise, significant improvements are observed in fine-tuned models. We conducted a thorough analysis of truth features using probes. Our visualization results show that orthogonal probes capture complementary truth-related features, forming well-defined clusters that reveal the inherent structure of the dataset.