# MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection

arXiv: https://arxiv.org/abs/2605.30288 · HTML: https://mlsanity.com/paper/2605.30288/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 26 of 26)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [QuRating: Selecting High-Quality Data for Training Language Models](https://mlsanity.com/paper/2402.09739/) (2024)
- [MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation](https://mlsanity.com/paper/2208.08227/) (2022)
- [OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling](https://mlsanity.com/paper/2506.20512/) (2025)
- [MoDS: Model-oriented Data Selection for Instruction Tuning](https://mlsanity.com/paper/2311.15653/) (2023)
- [DsDm: Model-Aware Dataset Selection with Datamodels](https://mlsanity.com/paper/2401.12926/) (2024)
- [Improving Pretraining Data Using Perplexity Correlations](https://mlsanity.com/paper/2409.05816/) (2024)
- [One-Shot Learning as Instruction Data Prospector for Large Language Models](https://mlsanity.com/paper/2312.10302/) (2023)
- [Large-Scale Data Selection for Instruction Tuning](https://mlsanity.com/paper/2503.01807/) (2025)
- [G-DIG: Towards Gradient-based Diverse and High-quality Instruction Data Selection for Machine Translation](https://mlsanity.com/paper/2405.12915/) (2024)
- [CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models](https://mlsanity.com/paper/2407.17467/) (2024)
- [Predictive Data Selection: The Data That Predicts Is the Data That Teaches](https://mlsanity.com/paper/2503.00808/) (2025)
- [A Survey on LLM Mid-Training](https://mlsanity.com/paper/2510.23081/) (2025)
- [Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training](https://mlsanity.com/paper/2411.14318/) (2024)
- [Midtraining Bridges Pretraining and Posttraining Distributions](https://mlsanity.com/paper/2510.14865/) (2025)
- [Enhancing Data Quality in Federated Fine-Tuning of Foundation Models](https://mlsanity.com/paper/2403.04529/) (2024)
- [Mining Useful General Data for Low-Resource Domain Adaptation](https://mlsanity.com/paper/2511.07380/) (2025)

## Built on by (most cited first, 3 of 3)

- [ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement](https://mlsanity.com/paper/2609.14857/) (2026)
- [RSI-Master: Structuring Experiments to Guide Autonomous Model Improvement](https://mlsanity.com/paper/2609.35561/) (2026)
- [AutoDataBench: A Data-centric Testbed for Accelerating Auto Research](https://mlsanity.com/paper/2609.40097/) (2026)

## Abstract (arXiv)

Mid-training has become an important stage in modern LLM development, using large-scale curated mixtures to strengthen capabilities before final post-training. Its data selection problem is distinct: the data are optimized under a pretraining-style objective at near-pretraining scale, but are curated toward downstream capabilities and drawn from heterogeneous sources with different formats and training roles. As a result, effective selection requires both scalability and source-adaptive semantic criteria. Existing model-based methods scale well, but provide only implicit quality signals. Semantic selection methods offer stronger judgments, but usually assume fixed rubrics or standardized data formats. To address this mismatch, we propose MIRA, a source-aware filtering framework based on self-anchored rubric discovery. The key idea is to make rubric construction part of data selection: MIRA first discovers what should be evaluated for each source group, then distills those judgments into scalable student scorers for full-corpus filtering. On code-oriented mid-training with 21 sources and 5 source groups, MIRA outperforms selection baselines across nine code benchmarks and matches the full-corpus run while using only half the tokens.