mlsanityData to 14 Oct 2026

GLM-130B: An Open Bilingual Pre-trained Model

arXiv 2210.02414 ↗First posted 5 Oct 2022cs.CL cs.AI cs.LGcited by 843

Not on the map. Why?

Cited by 75 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters. It is an attempt to open-source a 100B-scale model at least as good as GPT-3 (davinci) and unveil how models of such a scale can be successfully pre-trained. Over the course of this effort, we face numerous unexpected technical and engineering challenges, particularly on loss spikes and divergence. In this paper, we introduce the training process of GLM-130B including its design choices, training strategies for both efficiency and stability, and engineering efforts. The resultant GLM-130B model offers significant outperformance over GPT-3 175B (davinci) on a wide range of popular English benchmarks while the performance advantage is not observed in OPT-175B and BLOOM-176B. It also consistently and significantly outperforms ERNIE TITAN 3.0 260B -- the largest Chinese language model -- across related benchmarks. Finally, we leverage a unique scaling property of GLM-130B to reach INT4 quantization without post training, with almost no performance loss, making it the first among 100B-scale models and more importantly, allowing its effective inference on 4×RTX 3090 (24G) or 8×RTX 2080 Ti (11G) GPUs, the most affordable GPUs required for using 100B-scale models. The GLM-130B model weights are publicly accessible and its code, training logs, related toolkit, and lessons learned are open-sourced at urlhttps://github.com/THUDM/GLM-130B/.

Key facts. Cited by 843 papers in the map (data through 14 Oct 2026). Builds on 105 of its 154 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

105 of 154 references.

References · 105

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2016Layer Normalizationcited by 5,669
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
  13. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  14. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  15. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  16. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  17. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  18. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  19. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  20. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  21. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  22. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  23. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  24. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  25. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  26. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  27. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  28. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  29. 2017Mixed Precision Trainingcited by 1,156
  30. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  31. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  32. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  33. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  34. 2021Ethical and social risks of harm from Language Modelscited by 966
  35. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  36. 2020On Layer Normalization in the Transformer Architecturecited by 922
  37. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  38. 2019Are Sixteen Heads Really Better than One?cited by 816

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 383; 2024: 342; 2025: 76; 2026: 28.

843 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
38 more
  1. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  2. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  5. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  6. 2023BloombergGPT: A Large Language Model for Financecited by 939
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  8. 2023StarCoder: may the source be with you!cited by 799
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  10. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  13. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  14. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  15. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  16. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  17. 2023A Comprehensive Overview of Large Language Modelscited by 648
  18. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  19. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  20. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  21. 2024Large Language Models: A Surveycited by 525
  22. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  23. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  24. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  25. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  26. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  27. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  28. 2023Scaling Data-Constrained Language Modelscited by 403
  29. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  30. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  31. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  32. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  33. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  34. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  35. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  36. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  37. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  38. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300

50 most cited of 843 All 843 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GEB-1.3B: Open Lightweight Large Language Model
  2. 2025Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
  3. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale
  4. 2025Optimizing Large Language Model Training Using FP4 Quantization
  5. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
  6. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities
  7. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models
  8. 2024Stable LM 2 1.6B Technical Report