mlsanityData to 14 Oct 2026

Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Language

arXiv 2603.11881 ↗First posted 12 Mar 2026cs.CL cs.AIcited by 4

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

This report details the creation of Bielik-Minitron-7B, a compressed 7.35B parameter version of the Bielik-11B-v3.0 model, specifically optimized for European languages. By leveraging a two-stage compression methodology inspired by the NVIDIA Minitron approach, we combined structured hybrid pruning and knowledge distillation to reduce the model's parameter count by 33.4%, from 11.04B to 7.35B. We utilized the NVIDIA Model Optimizer for structural pruning and the NVIDIA NeMo Framework for logit-based distillation for quality recovery. Following distillation, the model underwent a rigorous alignment pipeline consisting of Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO-P), and Reinforcement Learning (GRPO). Our final model successfully recovered approximately 90% of the baseline model's performance while providing up to 50% inference speedup. This approach demonstrates an efficient pathway to create language models for less-represented languages, preserving the original model quality while reducing inference deployment costs.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 19 of its 28 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

19 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Compact Language Models via Pruning and Knowledge Distillation19 Jul

Other references · 18

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  7. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  8. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  9. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  10. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  11. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  12. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
6 more
  1. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 158
  2. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  3. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
  4. 2024Nemotron-4 15B Technical Reportcited by 36
  5. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 22
  6. 2024Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Seriescited by 1
  2. 2026Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scalecited by 1
  3. 2026Graded Entity-Familiarity Readouts in Language Models: Polish Adaptation, Cross-Language Robustness, and Refusal Steeringcited by 0
  4. 2026Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bitcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
  2. 2025Bielik 11B v3: Multilingual Large Language Model for European Languages
  3. 2026UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization, and Distillation
  4. 2025A Systematic Study of Compression Ordering for Large Language Models
  5. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models
  6. 2026Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guarantees
  7. 2024Model Compression and Efficient Inference for Large Language Models: A Survey
  8. 2025Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning