# AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis

arXiv: https://arxiv.org/abs/2610.13627 · HTML: https://mlsanity.com/paper/2610.13627/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 33 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 13 of 13)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs](https://mlsanity.com/paper/2307.03393/) (2023)
- [Realistic Synthetic Financial Transactions for Anti-Money Laundering Models](https://mlsanity.com/paper/2306.16424/) (2023)
- [Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps](https://mlsanity.com/paper/2502.14829/) (2025)
- [A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation](https://mlsanity.com/paper/2406.07320/) (2024)
- [Efficient Knowledge Graph Accuracy Evaluation](https://mlsanity.com/paper/1907.09657/) (2019)
- [Graph Feature Preprocessor: Real-time Subgraph-based Feature Extraction for Financial Crime Detection](https://mlsanity.com/paper/2402.08593/) (2024)
- [Exploring the In-Context Learning Capabilities of LLMs for Money Laundering Detection in Financial Graphs](https://mlsanity.com/paper/2507.14785/) (2025)

## Abstract (arXiv)

Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph prompt. Sampling reduces this cost, but small balanced subsets change unweighted precision and F1. We introduce AMLworld-Compact, an evaluation set that retains every illicit test transaction and samples benign transactions from supervised-score strata. Its HT-Coreset protocol reduces evaluation targets by 271x and 611x on HI-Small and LI-Small. Horvitz-Thompson weights give unbiased full-test confusion-count estimates for fixed predictions; recall is exact, while precision and F1 are ratio estimates. In a held-out study using Graph Feature Preprocessor (GFP) features, one boosted-tree family defines strata for evaluating the other over 500 draws per configuration. At the released sizes, F1 RMSE is 0.00-2.58 percentage points versus 4.44-7.41 for uniform sampling with HT weights; conservative intervals remain broad. Under the evaluated graph prompts, seven open-weight LLMs have weighted precision near the predict-all-illicit baseline. In an outcome-stratified audit of 1,000 HI-Small few-shot outputs from one run per model, lexical checks pass for 84.5% of correct and 56.2% of incorrect binary verdicts, so these markers do not establish correctness. We release the evaluation sets, construction code, and audit tools at https://huggingface.co/datasets/typhoon-ai/AMLworldCompactEval and https://github.com/nat-nischw/AMLworld-Compact.