# Bielik 11B v2 Technical Report

arXiv: https://arxiv.org/abs/2505.02410 · HTML: https://mlsanity.com/paper/2505.02410/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 5 May 2025.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive](https://mlsanity.com/paper/2402.13228/) (2024)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [Simple and Scalable Strategies to Continually Pre-train Large Language Models](https://mlsanity.com/paper/2403.08763/) (2024)
- [Model Stock: All we need is just a few fine-tuned models](https://mlsanity.com/paper/2403.19522/) (2024)
- [EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models](https://mlsanity.com/paper/2312.06281/) (2023)
- [MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures](https://mlsanity.com/paper/2406.06565/) (2024)
- [Continual Learning with Foundation Models: An Empirical Study of Latent Replay](https://mlsanity.com/paper/2205.00329/) (2022)
- [KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://mlsanity.com/paper/2005.00630/) (2020)
- [Instruction Tuning With Loss Over Instructions](https://mlsanity.com/paper/2405.14394/) (2024)
- [Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers](https://mlsanity.com/paper/2305.14858/) (2023)
- [Evaluation of Sentence Representations in Polish](https://mlsanity.com/paper/1910.11834/) (2019)
- [ScandEval: A Benchmark for Scandinavian Natural Language Processing](https://mlsanity.com/paper/2304.00906/) (2023)
- [StyloMetrix: An Open-Source Multilingual Tool for Representing Stylometric Vectors](https://mlsanity.com/paper/2309.12810/) (2023)
- [BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism](https://mlsanity.com/paper/2412.17933/) (2024)
- [Training Effective Neural Sentence Encoders from Automatically Mined Paraphrases](https://mlsanity.com/paper/2207.12759/) (2022)

## Built on by (most cited first, 2 of 2)

- [Making Bielik LLM Reason (Better): A Field Report](https://mlsanity.com/paper/2603.10640/) (2026)
- [sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak](https://mlsanity.com/paper/2610.09152/) (2026)

## Abstract (arXiv)

We present Bielik 11B v2, a state-of-the-art language model optimized for Polish text processing. Built on the Mistral 7B v0.2 architecture and scaled to 11B parameters using depth up-scaling, this model demonstrates exceptional performance across Polish language benchmarks while maintaining strong cross-lingual capabilities. We introduce two key technical innovations: Weighted Instruction Cross-Entropy Loss, which optimizes learning across diverse instruction types by assigning quality-based weights to training examples, and Adaptive Learning Rate, which dynamically adjusts based on context length. Comprehensive evaluation across multiple benchmarks demonstrates that Bielik 11B v2 outperforms many larger models, including those with 2-6 times more parameters, and significantly surpasses other specialized Polish language models on tasks ranging from linguistic understanding to complex reasoning. The model's parameter efficiency and extensive quantization options enable deployment across various hardware configurations, advancing Polish language AI capabilities and establishing new benchmarks for resource-efficient language modeling in less-represented languages.