# StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation

arXiv: https://arxiv.org/abs/2408.03281 · HTML: https://mlsanity.com/paper/2408.03281/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Builds on (most cited first, 29 of 29)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://mlsanity.com/paper/2311.04850/) (2023)
- [Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases](https://mlsanity.com/paper/2106.09231/) (2021)
- [E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT](https://mlsanity.com/paper/1911.03681/) (2019)
- [Wider and Deeper LLM Networks are Fairer LLM Evaluators](https://mlsanity.com/paper/2308.01862/) (2023)
- [Investigating Data Contamination for Pre-training Language Models](https://mlsanity.com/paper/2401.06059/) (2024)
- [Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting](https://mlsanity.com/paper/2311.13314/) (2023)
- [Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation](https://mlsanity.com/paper/2306.05783/) (2023)
- [Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View](https://mlsanity.com/paper/2203.12258/) (2022)
- [Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models](https://mlsanity.com/paper/2305.09144/) (2023)
- [Ask Again, Then Fail: Large Language Models' Vacillations in Judgment](https://mlsanity.com/paper/2310.02174/) (2023)

## Built on by (most cited first, 9 of 9)

- [Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation](https://mlsanity.com/paper/2502.17521/) (2025)
- [Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers](https://mlsanity.com/paper/2410.11672/) (2024)
- [Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languages](https://mlsanity.com/paper/2601.06341/) (2026)
- [ReportLogic: Evaluating Logical Quality in Deep Research Reports](https://mlsanity.com/paper/2602.18446/) (2026)
- [Investigating Advanced Reasoning of Large Language Models via Black-Box Environment Interaction](https://mlsanity.com/paper/2508.19035/) (2025)
- [Automated Safety Benchmarking: A Multi-agent Pipeline for LVLMs](https://mlsanity.com/paper/2601.19507/) (2026)
- [From Lemmas to Dependencies: What Signals Drive Light Verbs Classification?](https://mlsanity.com/paper/2602.04127/) (2026)
- [CienaLLM: Generative Climate-Impact Extraction from News Articles with Autoregressive LLMs](https://mlsanity.com/paper/2512.19305/) (2025)
- [RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving](https://mlsanity.com/paper/2609.00062/) (2026)

## Abstract (arXiv)

Evaluation is the baton for the development of large language models. Current evaluations typically employ a single-item assessment paradigm for each atomic test objective, which struggles to discern whether a model genuinely possesses the required capabilities or merely memorizes/guesses the answers to specific questions. To this end, we propose a novel evaluation framework referred to as StructEval. Starting from an atomic test objective, StructEval deepens and broadens the evaluation by conducting a structured assessment across multiple cognitive levels and critical concepts, and therefore offers a comprehensive, robust and consistent evaluation for LLMs. Experiments on three widely-used benchmarks demonstrate that StructEval serves as a reliable tool for resisting the risk of data contamination and reducing the interference of potential biases, thereby providing more reliable and consistent conclusions regarding model capabilities. Our framework also sheds light on the design of future principled and trustworthy LLM evaluation protocols.