mlsanityData to 14 Oct 2026

360Zhinao Technical Report

arXiv 2405.13386 ↗First posted 22 May 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We present 360Zhinao models with 7B parameter size and context lengths spanning 4K, 32K and 360K, all available at https://github.com/Qihoo360/360zhinao. For rapid development in pretraining, we establish a stable and sensitive ablation environment to evaluate and compare experiment runs with minimal model size. Under such guidance, we perfect our data cleaning and composition strategies to pretrain texttt360Zhinao-7B-Base on 3.4T tokens. We also mainly emphasize data during alignment, where we strive to balance quantity and quality with filtering and reformatting. With tailored data, 360Zhinao-7B's context window is easily extended to 32K and 360K. RMs and RLHF are trained following SFT and credibly applied to specific tasks. All together these contributions lead to 360Zhinao-7B's competitive performance among models of similar size.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 71 of its 82 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

71 of 82 references.

References · 71

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
38 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2023A Survey of Large Language Modelscited by 2,576
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2020GLU Variants Improve Transformercited by 1,804
  12. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  13. 2019Root Mean Square Layer Normalizationcited by 1,480
  14. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  15. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  16. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  17. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  18. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  19. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  20. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  21. 2024Yi: Open Foundation Models by 01.AIcited by 719
  22. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  23. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  24. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  25. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  26. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  27. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  28. 2023Textbooks Are All You Needcited by 569
  29. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  30. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  31. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  32. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  33. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  34. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  35. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  36. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  37. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  38. 2023Effective Long-Context Scaling of Foundation Modelscited by 303

50 most cited of 71 All 71 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Pre-training under infinite compute
  2. 2025Improving Quantization with Post-Training Model Expansion
  3. 2026Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inference
  4. 2026ArXiv-to-Model: A Practical Study of Scientific LM Training
  5. 2025Reproducibility Report: Test-Time Training on Nearest Neighbors for Large Language Models
  6. 2025Characterization and Mitigation of Training Instabilities in Microscaling Formats
  7. 2026Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
  8. 2024Lillama: Large Language Models Compression via Low-Rank Feature Distillation