mlsanityData to 14 Oct 2026

GECKO: Generative Language Model for English, Code and Korean

arXiv 2405.15640 ↗First posted 24 May 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce GECKO, a bilingual large language model (LLM) optimized for Korean and English, along with programming languages. GECKO is pretrained on the balanced, high-quality corpus of Korean and English employing LLaMA architecture. In this report, we share the experiences of several efforts to build a better data pipeline for the corpus and to train our model. GECKO shows great efficiency in token generations for both Korean and English, despite its small size of vocabulary. We measure the performance on the representative benchmarks in terms of Korean, English and Code, and it exhibits great performance on KMMLU (Korean MMLU) and modest performance in English and Code, even with its smaller number of trained tokens compared to English-focused LLMs. GECKO is available to the open-source community under a permissive license. We hope our work offers a research baseline and practical insights for Korean LLM research. The model can be found at: https://huggingface.co/kifai/GECKO-7B

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

34 of 49 references.

References · 34

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023Mistral 7Bcited by 3,164
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
22 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2024Mixtral of Expertscited by 1,823
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  5. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  8. 2024Yi: Open Foundation Models by 01.AIcited by 719
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  11. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  12. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  13. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  14. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  15. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  16. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  17. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  18. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  19. 2024Large language models surpass human experts in predicting neuroscience resultscited by 69
  20. 2023Typhoon: Thai Large Language Modelscited by 30
  21. 2023A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Modelscited by 22
  22. 2024Poro 34B and the Blessing of Multilingualitycited by 18

Built on by

Not cited yet.