# Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation

arXiv: https://arxiv.org/abs/2410.18565 · HTML: https://mlsanity.com/paper/2410.18565/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 57 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [Orca-Math: Unlocking the potential of SLMs in Grade School Math](https://mlsanity.com/paper/2402.14830/) (2024)
- [Simple and Scalable Strategies to Continually Pre-train Large Language Models](https://mlsanity.com/paper/2403.08763/) (2024)
- [Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations](https://mlsanity.com/paper/2207.10050/) (2022)
- [Continual Learning with Foundation Models: An Empirical Study of Latent Replay](https://mlsanity.com/paper/2205.00329/) (2022)
- [KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://mlsanity.com/paper/2005.00630/) (2020)
- [Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training](https://mlsanity.com/paper/2006.09092/) (2020)
- [Instruction Tuning With Loss Over Instructions](https://mlsanity.com/paper/2405.14394/) (2024)
- [Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers](https://mlsanity.com/paper/2305.14858/) (2023)
- [Evaluation of Sentence Representations in Polish](https://mlsanity.com/paper/1910.11834/) (2019)
- [StyloMetrix: An Open-Source Multilingual Tool for Representing Stylometric Vectors](https://mlsanity.com/paper/2309.12810/) (2023)
- [Training Effective Neural Sentence Encoders from Automatically Mined Paraphrases](https://mlsanity.com/paper/2207.12759/) (2022)
- [PolQA: Polish Question Answering Dataset](https://mlsanity.com/paper/2212.08897/) (2022)
- [Extending the Subwording Model of Multilingual Pretrained Models for New Languages](https://mlsanity.com/paper/2211.15965/) (2022)

## Built on by (most cited first, 15 of 15)

- [Krikri: Advancing Open Large Language Models for Greek](https://mlsanity.com/paper/2505.13772/) (2025)
- [Evaluating Polish linguistic and cultural competency in large language models](https://mlsanity.com/paper/2503.00995/) (2025)
- [Model-Aware Tokenizer Transfer](https://mlsanity.com/paper/2510.21954/) (2025)
- [Bielik 11B v3: Multilingual Large Language Model for European Languages](https://mlsanity.com/paper/2601.11579/) (2025)
- [Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Language](https://mlsanity.com/paper/2603.11881/) (2026)
- [Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models](https://mlsanity.com/paper/2506.07645/) (2025)
- [PL-Guard: Benchmarking Language Model Safety for Polish](https://mlsanity.com/paper/2506.16322/) (2025)
- [Bielik Guard: Efficient Polish Language Safety Classifiers for LLM Content Moderation](https://mlsanity.com/paper/2602.07954/) (2026)
- [Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series](https://mlsanity.com/paper/2604.10799/) (2026)
- [Writing-System-Level Tokenizer Adaptation for Byte-Level BPE](https://mlsanity.com/paper/2608.00582/) (2026)
- [Integrating gender inclusivity into large language models via instruction tuning](https://mlsanity.com/paper/2508.18466/) (2025)
- [Making Bielik LLM Reason (Better): A Field Report](https://mlsanity.com/paper/2603.10640/) (2026)
- [The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs](https://mlsanity.com/paper/2606.07422/) (2026)
- [Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bit](https://mlsanity.com/paper/2609.32042/) (2026)
- [Wieszcz-XIX: A 3.1-Billion-Word Corpus of Pre-1918 Polish and Temporally Bounded Language Models Trained From Scratch](https://mlsanity.com/paper/2610.10592/) (2026)

## Abstract (arXiv)

We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for Polish language processing. Trained on curated Polish corpora, this model addresses key challenges in language model development through innovative techniques. These include Weighted Instruction Cross-Entropy Loss, which balances the learning of different instruction types, and Adaptive Learning Rate, which dynamically adjusts the learning rate based on training progress. To evaluate performance, we created the Open PL LLM Leaderboard and Polish MT-Bench, novel frameworks assessing various NLP tasks and conversational abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9 percentage point increase in average score compared to Mistral-7B-v0.1 on the RAG Reader task. It also excels in the Polish MT-Bench, particularly in Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model represents a substantial advancement in Polish language AI, offering a powerful tool for diverse linguistic applications and setting new benchmarks in the field.