# Learning More from Less: Unlocking Internal Representations for Benchmark Compression

arXiv: https://arxiv.org/abs/2602.00710 · HTML: https://mlsanity.com/paper/2602.00710/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 31 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations](https://mlsanity.com/paper/2410.02707/) (2024)
- [Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification](https://mlsanity.com/paper/2504.05419/) (2025)
- [SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension](https://mlsanity.com/paper/2404.16790/) (2024)
- [DeepSeek-R1 Thoughtology: Let's think about LLM Reasoning](https://mlsanity.com/paper/2504.07128/) (2025)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations](https://mlsanity.com/paper/2509.12886/) (2025)
- [Improving Model Evaluation using SMART Filtering of Benchmark Datasets](https://mlsanity.com/paper/2410.20245/) (2024)
- [Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation](https://mlsanity.com/paper/2502.13576/) (2025)
- [Large Language Models Have Intrinsic Meta-Cognition, but Need a Good Lens](https://mlsanity.com/paper/2506.08410/) (2025)
- [EffiEval: Efficient and Generalizable Model Evaluation via Capability Coverage Maximization](https://mlsanity.com/paper/2508.09662/) (2025)
- [LLMs Encode How Difficult Problems Are](https://mlsanity.com/paper/2510.18147/) (2025)

## Built on by (most cited first, 2 of 2)

- [LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering](https://mlsanity.com/paper/2608.28281/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)

## Abstract (arXiv)

The prohibitive cost of evaluating Large Language Models (LLMs) necessitates efficient alternatives to full-scale benchmarking. Prevalent approaches address this by identifying a small coreset of items to approximate full-benchmark performance. However, existing methods must estimate a reliable item profile from response patterns across many source models, which becomes statistically unstable when the source pool is small. This dependency is particularly limiting for newly released benchmarks with minimal historical evaluation data. We argue that discrete correctness labels are a lossy view of the model's decision process and fail to capture information encoded in hidden states. To address this, we introduce RepCore, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets. Using these subsets for performance extrapolation, RepCore achieves precise estimation accuracy with as few as ten source models. Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy. Spectral analysis further indicates that the aligned representations contain separable components reflecting broad response tendencies and task-specific reasoning patterns.