mlsanityData to 7 Oct 2026

Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca

arXiv 2304.08177 ↗First posted 17 Apr 2023cs.CL cs.HC cs.LGcited by 302

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs), such as ChatGPT and GPT-4, have dramatically transformed natural language processing research and shown promising strides towards Artificial General Intelligence (AGI). Nonetheless, the high costs associated with training and deploying LLMs present substantial obstacles to transparent, accessible academic research. While several large language models, such as LLaMA, have been open-sourced by the community, these predominantly focus on English corpora, limiting their usefulness for other languages. In this paper, we propose a method to augment LLaMA with capabilities for understanding and generating Chinese text and its ability to follow instructions. We achieve this by extending LLaMA's existing vocabulary with an additional 20,000 Chinese tokens, thereby improving its encoding efficiency and semantic understanding of Chinese. We further incorporate secondary pre-training using Chinese data and fine-tune the model with Chinese instruction datasets, significantly enhancing the model's ability to comprehend and execute instructions. Our experimental results indicate that the newly proposed model markedly enhances the original LLaMA's proficiency in understanding and generating Chinese content. Additionally, the results on the C-Eval dataset yield competitive performance among the models with several times the size of ours. We have made our pre-trained models, training scripts, and other resources available through GitHub, fostering open research for our community. Chinese LLaMA series: urlhttps://github.com/ymcui/Chinese-LLaMA-Alpaca and Chinese Llama-2 series: urlhttps://github.com/ymcui/Chinese-LLaMA-Alpaca-2

Key facts. Cited by 302 papers in the map (data through 7 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

19 of 34 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2020GLU Variants Improve Transformercited by 1,724
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
7 more
  1. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  5. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 262
  6. 2022CINO: A Chinese Minority Pre-trained Language Modelcited by 11
  7. 2022LERT: A Linguistically-motivated Pre-trained Language Modelcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 72; 2024: 147; 2025: 54; 2026: 29.

302 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  3. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  4. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  5. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  6. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  7. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  8. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  9. 2024How do Large Language Models Handle Multilingualism?cited by 148
  10. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 118
  11. 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 117
  12. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
38 more
  1. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 113
  2. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  3. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  4. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  5. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  6. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  7. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 89
  8. 2024Tuning Language Models by Proxycited by 88
  9. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 80
  10. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  11. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  12. 2023GPT Can Solve Mathematical Problems Without a Calculatorcited by 68
  13. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  14. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  15. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  16. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 59
  17. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  18. 2023Lawyer LLaMA Technical Reportcited by 54
  19. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 54
  20. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmarkcited by 54
  21. 2023FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Modelscited by 52
  22. 2023Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languagescited by 52
  23. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  24. 2023LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuningcited by 47
  25. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  26. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  27. 2024COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuningcited by 45
  28. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  29. 2024Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?cited by 40
  30. 2025Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Studycited by 38
  31. 2024LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Modelscited by 36
  32. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 34
  33. 2024LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Modelcited by 34
  34. 2024REEF: Representation Encoding Fingerprints for Large Language Modelscited by 34
  35. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  36. 2023HuRef: HUman-REadable Fingerprint for Large Language Modelscited by 32
  37. 2024ROSE Doesn't Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decodingcited by 32
  38. 2023Aligning Large Language Models with Human: A Surveycited by 31

50 most cited of 302 All 302 (JSON)