# CODEBLOCK: Learning to Supervise Code at the Right Granularity

arXiv: https://arxiv.org/abs/2606.18286 · HTML: https://mlsanity.com/paper/2606.18286/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Builds on (most cited first, 30 of 31)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [Rho-1: Not All Tokens Are What You Need](https://mlsanity.com/paper/2404.07965/) (2024)
- [OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models](https://mlsanity.com/paper/2411.04905/) (2024)
- [Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability](https://mlsanity.com/paper/2411.19943/) (2024)
- [SelfCodeAlign: Self-Alignment for Code Generation](https://mlsanity.com/paper/2410.24198/) (2024)
- [OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs](https://mlsanity.com/paper/2504.04030/) (2025)
- [Seed-Coder: Let the Code Model Curate Data for Itself](https://mlsanity.com/paper/2506.03524/) (2025)
- [Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning](https://mlsanity.com/paper/2502.01968/) (2025)
- [Improving Data Efficiency via Curating LLM-Driven Rating Systems](https://mlsanity.com/paper/2410.10877/) (2024)
- [Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric](https://mlsanity.com/paper/2502.17184/) (2025)
- [D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuning](https://mlsanity.com/paper/2503.11441/) (2025)
- [Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning](https://mlsanity.com/paper/2407.05040/) (2024)
- [T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning](https://mlsanity.com/paper/2506.01317/) (2025)
- [MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space](https://mlsanity.com/paper/2504.13835/) (2025)
- [ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning](https://mlsanity.com/paper/2412.00631/) (2024)
- [MSL: Not All Tokens Are What You Need for Tuning LLM as a Recommender](https://mlsanity.com/paper/2504.04178/) (2025)
- [SCAR: Data Selection via Style Consistency-Aware Response Ranking for Efficient Instruction-Tuning of Large Language Models](https://mlsanity.com/paper/2406.10882/) (2024)
- [Co-Learning: Code Learning for Multi-Agent Reinforcement Collaborative Framework with Conversational Natural Language Interfaces](https://mlsanity.com/paper/2409.00985/) (2024)
- [ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning](https://mlsanity.com/paper/2510.18250/) (2025)
- [LM-mixup: Text Data Augmentation via Language Model based Mixup](https://mlsanity.com/paper/2510.20449/) (2025)
- [Token-level Data Selection for Safe LLM Fine-tuning](https://mlsanity.com/paper/2603.01185/) (2026)

## Built on by (most cited first, 1 of 1)

- [AUV-Bench: Aesthetic Understanding and Generation Evaluation for User Interfaces](https://mlsanity.com/paper/2609.34854/) (2026)

## Abstract (arXiv)

Supervised fine-tuning of code LLMs typically applies uniform cross-entropy loss to all response tokens, implicitly assuming that every token provides equally useful learning signals. Recent token-level selection methods challenge this assumption in natural-language SFT by supervising only high-value tokens. However, such pointwise selection can fragment the syntactic structures and program dependencies of code, leaving supervision scattered across incomplete code units. In experiments on 30K code instruction-response pairs, under the same 10% token budget, supervising complete coding blocks improves average performance by 11.7 points over prior approaches that supervise isolated tokens. Motivated by this observation, we propose CodeBlock, a structure-aware sparse supervision framework that uses complete, parser-aligned coding items as the basic units of supervision. CodeBlock constructs coding items from high-quality code instruction data, estimates their supervision utility using GCE, which is more robust to low-probability tokens, and further adjusts their supervision priority using data-flow reach and bridge signals. During training, the full response is retained as context, while loss is applied only to the selected supervision units. Experiments show that partial supervision over only about 6.9% of response tokens consistently outperforms full-token SFT across all five model settings, suggesting that effective code SFT depends not only on identifying high-value supervision, but also on allocating it at the appropriate structural granularity.