mlsanityData to 7 Oct 2026

Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation

arXiv 2410.18565 ↗First posted 24 Oct 2024cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for Polish language processing. Trained on curated Polish corpora, this model addresses key challenges in language model development through innovative techniques. These include Weighted Instruction Cross-Entropy Loss, which balances the learning of different instruction types, and Adaptive Learning Rate, which dynamically adjusts the learning rate based on training progress. To evaluate performance, we created the Open PL LLM Leaderboard and Polish MT-Bench, novel frameworks assessing various NLP tasks and conversational abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9 percentage point increase in average score compared to Mistral-7B-v0.1 on the RAG Reader task. It also excels in the Polish MT-Bench, particularly in Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model represents a substantial advancement in Polish language AI, offering a powerful tool for diverse linguistic applications and setting new benchmarks in the field.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 29 of its 57 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

29 of 57 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2023Mistral 7Bcited by 3,065
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2020GLU Variants Improve Transformercited by 1,724
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
17 more
  1. 2016Language Modeling with Gated Convolutional Networkscited by 688
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  3. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  4. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  5. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  6. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  7. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  8. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  9. 2020KLEJ: Comprehensive Benchmark for Polish Language Understandingcited by 51
  10. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  11. 2024Instruction Tuning With Loss Over Instructionscited by 40
  12. 2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30
  13. 2019Evaluation of Sentence Representations in Polishcited by 18
  14. 2023StyloMetrix: An Open-Source Multilingual Tool for Representing Stylometric Vectorscited by 14
  15. 2022Training Effective Neural Sentence Encoders from Automatically Mined Paraphrasescited by 11
  16. 2022PolQA: Polish Question Answering Datasetcited by 7
  17. 2022Extending the Subwording Model of Multilingual Pretrained Models for New Languagescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025Krikri: Advancing Open Large Language Models for Greekcited by 13
  2. 2025Evaluating Polish linguistic and cultural competency in large language modelscited by 6
  3. 2025Bielik 11B v3: Multilingual Large Language Model for European Languagescited by 6
  4. 2025Model-Aware Tokenizer Transfercited by 5
  5. 2026Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Languagecited by 4
  6. 2025Evaluating LLMs Robustness in Less Resourced Languages with Proxy Modelscited by 2
  7. 2025PL-Guard: Benchmarking Language Model Safety for Polishcited by 2
  8. 2026Bielik Guard: Efficient Polish Language Safety Classifiers for LLM Content Moderationcited by 2
  9. 2026Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Seriescited by 1
  10. 2026Writing-System-Level Tokenizer Adaptation for Byte-Level BPEcited by 1
  11. 2025Integrating gender inclusivity into large language models via instruction tuningcited by 0
  12. 2026Making Bielik LLM Reason (Better): A Field Reportcited by 0
2 more
  1. 2026The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMscited by 0
  2. 2026Quantization Thresholds Replicate, Failure Modes Do Not: A Three-Model Study of Agentic Tool Use in Polish from 8-bit to 2-bitcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PLLuM: A Family of Polish Large Language Models
  2. 2024Evaluating Large Language Models with Human Feedback: Establishing a Swedish Benchmark
  3. 2026sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak
  4. 2023An Automated Survey of Generative Artificial Intelligence: Large Language Models, Architectures, Protocols, and Applications
  5. 2023NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian
  6. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  7. 2025GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture
  8. 2025Optimising Language Models for Downstream Tasks: A Post-Training Perspective