# CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets

arXiv: https://arxiv.org/abs/2610.07132 · HTML: https://mlsanity.com/paper/2610.07132/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 59 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 8 of 8)

- [Efficient Guided Generation for Large Language Models](https://mlsanity.com/paper/2307.09702/) (2023)
- [ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction](https://mlsanity.com/paper/2602.12247/) (2026)
- [A Standardized Machine-readable Dataset Documentation Format for Responsible AI](https://mlsanity.com/paper/2407.16883/) (2024)
- [Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability](https://mlsanity.com/paper/2506.01789/) (2025)
- [LLMStructBench: Benchmarking Large Language Model Structured Data Extraction](https://mlsanity.com/paper/2602.14743/) (2026)
- [Do Not Trust Licenses You See: Dataset Compliance Requires Massive-Scale AI-Powered Lifecycle Tracing](https://mlsanity.com/paper/2503.02784/) (2025)
- [Schema as Parameterized Tools for Universal Information Extraction](https://mlsanity.com/paper/2506.01276/) (2025)
- [MeXtract: Light-Weight Metadata Extraction from Scientific Papers](https://mlsanity.com/paper/2510.06889/) (2025)

## Abstract (arXiv)

Croissant has emerged as a standard for machine-readable dataset metadata, yet populating its fields remains labor-intensive and requires careful reading of accompanying dataset documentation. We present the first benchmark enabling end-to-end evaluation of metadata extraction aligned with a community-standard schema. The benchmark comprises 602 papers, including 102 with human-validated gold annotations and 500 with LLM-generated silver annotations, covering the full Croissant schema with both core and Responsible AI (RAI) fields. Using this benchmark, we evaluate a range of extraction systems spanning frontier models, open-weight models, and agentic architectures, under a two-tier evaluation framework that combines rule-based scoring with an LLM judge selected via human audit. We find that single-pass extraction consistently outperforms the four agentic architectures we evaluate: across backbones, these decomposed variants achieve lower accuracy than a single full-context pass. The largest gap appears on long-form RAI fields, which require synthesizing and interpreting information scattered across a paper rather than copying it from a single location, a setting where current systems remain far from reliable. We release the benchmark, evaluation code, judge audit, a live demo, and a leaderboard open to new systems.