# A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset

arXiv: https://arxiv.org/abs/1806.00358 · HTML: https://mlsanity.com/paper/1806.00358/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2018.

## Built on by (most cited first, 30 of 55)

- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [ScienceWorld: Is your Agent Smarter than a 5th Grader?](https://mlsanity.com/paper/2203.07540/) (2022)
- [MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices](https://mlsanity.com/paper/2312.16886/) (2023)
- [OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting](https://mlsanity.com/paper/2501.13987/) (2025)
- [Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge](https://mlsanity.com/paper/2102.03315/) (2021)
- [Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression](https://mlsanity.com/paper/2403.15447/) (2024)
- [Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization](https://mlsanity.com/paper/2504.09629/) (2025)
- [QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models](https://mlsanity.com/paper/2310.09259/) (2023)
- [Efficient Continual Pre-training for Building Domain Specific Large Language Models](https://mlsanity.com/paper/2311.08545/) (2023)
- [ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization](https://mlsanity.com/paper/2406.05981/) (2024)
- [MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods](https://mlsanity.com/paper/2501.13484/) (2025)
- [Can Large Language Models Understand Context?](https://mlsanity.com/paper/2402.00858/) (2024)
- [I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models](https://mlsanity.com/paper/2405.17849/) (2024)
- [Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation](https://mlsanity.com/paper/2506.17088/) (2025)
- [Pruning Foundation Models for High Accuracy without Retraining](https://mlsanity.com/paper/2410.15567/) (2024)
- [Block Rotation is All You Need for MXFP4 Quantization](https://mlsanity.com/paper/2511.04214/) (2025)
- [EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs](https://mlsanity.com/paper/2403.02775/) (2024)
- [DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization](https://mlsanity.com/paper/2511.04063/) (2025)
- [One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models](https://mlsanity.com/paper/2310.09499/) (2023)
- [MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design](https://mlsanity.com/paper/2412.14590/) (2024)
- [ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks](https://mlsanity.com/paper/2312.08583/) (2023)
- [Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers](https://mlsanity.com/paper/2211.11586/) (2022)
- [SpecQuant: Spectral Decomposition and Adaptive Truncation for Ultra-Low-Bit LLMs Quantization](https://mlsanity.com/paper/2511.11663/) (2025)
- [KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models](https://mlsanity.com/paper/2602.11184/) (2026)
- [DLP: Dynamic Layerwise Pruning in Large Language Models](https://mlsanity.com/paper/2505.23807/) (2025)
- [GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensing](https://mlsanity.com/paper/2604.08896/) (2026)
- [DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation](https://mlsanity.com/paper/2503.21135/) (2025)

## Abstract (arXiv)

The recent work of Clark et al. introduces the AI2 Reasoning Challenge (ARC) and the associated ARC dataset that partitions open domain, complex science questions into an Easy Set and a Challenge Set. That paper includes an analysis of 100 questions with respect to the types of knowledge and reasoning required to answer them; however, it does not include clear definitions of these types, nor does it offer information about the quality of the labels. We propose a comprehensive set of definitions of knowledge and reasoning types necessary for answering the questions in the ARC dataset. Using ten annotators and a sophisticated annotation interface, we analyze the distribution of labels across the Challenge Set and statistics related to them. Additionally, we demonstrate that although naive information retrieval methods return sentences that are irrelevant to answering the query, sufficient supporting text is often present in the (ARC) corpus. Evaluating with human-selected relevant sentences improves the performance of a neural machine comprehension model by 42 points.