# How to Get Your LLM to Generate Challenging Problems for Evaluation

arXiv: https://arxiv.org/abs/2502.14678 · HTML: https://mlsanity.com/paper/2502.14678/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2025.

## Built on by (most cited first, 11 of 11)

- [DeepSeek-R1 Thoughtology: Let's think about LLM Reasoning](https://mlsanity.com/paper/2504.07128/) (2025)
- [AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis](https://mlsanity.com/paper/2510.24695/) (2025)
- [Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator](https://mlsanity.com/paper/2505.20738/) (2025)
- [Executable Functional Abstractions: Inferring Generative Programs for Advanced Math Problems](https://mlsanity.com/paper/2504.09763/) (2025)
- [Automating Benchmark Design](https://mlsanity.com/paper/2510.25039/) (2025)
- [CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks](https://mlsanity.com/paper/2606.03650/) (2026)
- [STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator](https://mlsanity.com/paper/2604.24544/) (2026)
- [Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces](https://mlsanity.com/paper/2606.05464/) (2026)
- [VeriGeo: Controllable Geometry Question Generation with Numerical and Analytical Verification](https://mlsanity.com/paper/2606.14176/) (2026)
- [Ask-E: An Environment for Calibrated Question Generation](https://mlsanity.com/paper/2608.06933/) (2026)
- [You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments](https://mlsanity.com/paper/2609.04384/) (2026)

## Abstract (arXiv)

The pace of evolution of Large Language Models (LLMs) necessitates new approaches for rigorous and comprehensive evaluation. Traditional human annotation is increasingly impracticable due to the complexities and costs involved in generating high-quality, challenging problems. In this work, we introduce CHASE, a unified framework to synthetically generate challenging problems using LLMs without human involvement. For a given task, our approach builds a hard problem in a bottom-up manner from simpler components. Moreover, our framework decomposes the generation process into independently verifiable sub-tasks, thereby ensuring a high level of quality and correctness. We implement CHASE to create evaluation benchmarks across three diverse domains: (1) document-based question answering, (2) repository-level code completion, and (3) math reasoning. The performance of state-of-the-art LLMs on these synthetic benchmarks lies in the range of 40-60% accuracy, thereby demonstrating the effectiveness of our framework at generating challenging problems. We publicly release our benchmarks and code.