# Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations

arXiv: https://arxiv.org/abs/2508.03550 · HTML: https://mlsanity.com/paper/2508.03550/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 44 of its 62 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 44)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge](https://mlsanity.com/paper/2407.19594/) (2024)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Benchmarking Foundation Models with Language-Model-as-an-Examiner](https://mlsanity.com/paper/2306.04181/) (2023)
- [Generative Judge for Evaluating Alignment](https://mlsanity.com/paper/2310.05470/) (2023)
- [HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM](https://mlsanity.com/paper/2311.09528/) (2023)
- [Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning](https://mlsanity.com/paper/2402.00530/) (2024)
- [HelpSteer2-Preference: Complementing Ratings with Preferences](https://mlsanity.com/paper/2410.01257/) (2024)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [Preference Leakage: A Contamination Problem in LLM-as-a-judge](https://mlsanity.com/paper/2502.01534/) (2025)
- [Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement](https://mlsanity.com/paper/2407.18370/) (2024)
- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models](https://mlsanity.com/paper/2312.06281/) (2023)
- [Split and Merge: Aligning Position Biases in LLM-based Evaluators](https://mlsanity.com/paper/2310.01432/) (2023)
- [TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks](https://mlsanity.com/paper/2310.00752/) (2023)
- [Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs](https://mlsanity.com/paper/2410.13835/) (2024)
- [Measuring and Controlling Instruction (In)Stability in Language Model Dialogs](https://mlsanity.com/paper/2402.10962/) (2024)
- [ICE-Score: Instructing Large Language Models to Evaluate Code](https://mlsanity.com/paper/2304.14317/) (2023)
- [Are LLM-based Evaluators Confusing NLG Quality Criteria?](https://mlsanity.com/paper/2402.12055/) (2024)
- [Rethinking Data Selection for Supervised Fine-Tuning](https://mlsanity.com/paper/2402.06094/) (2024)
- [Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions](https://mlsanity.com/paper/2408.08781/) (2024)
- [HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition](https://mlsanity.com/paper/2402.15754/) (2024)
- [Beyond Scalar Reward Model: Learning Generative Judge from Preference Data](https://mlsanity.com/paper/2410.03742/) (2024)
- [West-of-N: Synthetic Preferences for Self-Improving Reward Models](https://mlsanity.com/paper/2401.12086/) (2024)
- [When Backdoors Speak: Understanding LLM Backdoor Attacks Through Model-Generated Explanations](https://mlsanity.com/paper/2411.12701/) (2024)

## Built on by (most cited first, 15 of 15)

- [Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale](https://mlsanity.com/paper/2601.03444/) (2026)
- [LLM Safety From Within: Detecting Harmful Content with Internal Representations](https://mlsanity.com/paper/2604.18519/) (2026)
- [Enhancing Uncertainty Estimation in LLMs with Expectation of Aggregated Internal Belief](https://mlsanity.com/paper/2509.01564/) (2025)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [Zero-shot Large Language Models for Automatic Readability Assessment](https://mlsanity.com/paper/2604.24470/) (2026)
- [FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios](https://mlsanity.com/paper/2605.00706/) (2026)
- [GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models](https://mlsanity.com/paper/2605.01256/) (2026)
- [D-Judge: Disrupting Multi-Turn Jailbreaks using Semantics-Preserving Output Rewriting](https://mlsanity.com/paper/2606.02640/) (2026)
- [Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict](https://mlsanity.com/paper/2609.32407/) (2026)
- [Scope3Trace: Evidence-Based Identification and Extraction of Scope 3 GHG Emissions from Sustainability Reports](https://mlsanity.com/paper/2607.17122/) (2026)
- [AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection](https://mlsanity.com/paper/2609.05899/) (2026)
- [UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms](https://mlsanity.com/paper/2609.05910/) (2026)
- [ForeSight: Enhancing Risk Monitoring via Early Safety Signal Distillation](https://mlsanity.com/paper/2609.13737/) (2026)
- [Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Prompting](https://mlsanity.com/paper/2609.24650/) (2026)
- [Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations](https://mlsanity.com/paper/2610.07115/) (2026)

## Abstract (arXiv)

The growing scale of evaluation tasks has led to the widespread adoption of automated evaluation using LLMs, a paradigm known as "LLM-as-a-judge". However, improving its alignment with human preferences without complex prompts or fine-tuning remains challenging. Previous studies mainly optimize based on shallow outputs, overlooking rich cross-layer representations. In this work, motivated by preliminary findings that middle-to-upper layers encode semantically and task-relevant representations that are often more aligned with human judgments than the final layer, we propose LAGER, a post-hoc, plug-and-play framework for improving the alignment of LLM-as-a-Judge point-wise evaluations with human scores by leveraging internal representations. LAGER produces fine-grained judgment scores by aggregating cross-layer score-token logits and computing the expected score from a softmax-based distribution, while keeping the LLM backbone frozen and ensuring no impact on the inference process. LAGER fully leverages the complementary information across different layers, overcoming the limitations of relying solely on the final layer. We evaluate our method on the standard alignment benchmarks Flask, HelpSteer, and BIGGen using Spearman correlation, and find that LAGER achieves improvements of up to 7.5% over the best baseline across these benchmarks. Without reasoning steps, LAGER matches or outperforms reasoning-based methods. Experiments on downstream applications, such as data selection and emotional understanding, further show the generalization of LAGER.