# The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes

arXiv: https://arxiv.org/abs/2601.11659 · HTML: https://mlsanity.com/paper/2601.11659/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 11 references that are in the map.
- First posted on arXiv 15 Jan 2026.

## Built on by (most cited first, 30 of 33)

- [LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers](https://mlsanity.com/paper/2608.06867/) (2026)
- [Tucano 2 Cool: Better Open Source LLMs for Portuguese](https://mlsanity.com/paper/2603.03543/) (2026)
- [Learning to Wander: Improving the Global Image Geolocation Ability of LMMs via Actionable Reasoning](https://mlsanity.com/paper/2603.10463/) (2026)
- [Are VLMs Lost Between Sky and Space? LinkS2Bench for UAV-Satellite Dynamic Cross-View Spatial Intelligence](https://mlsanity.com/paper/2604.02020/) (2026)
- [An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence](https://mlsanity.com/paper/2602.02400/) (2026)
- [Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM](https://mlsanity.com/paper/2605.05927/) (2026)
- [ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models](https://mlsanity.com/paper/2605.23694/) (2026)
- [Organizational Principles Enable Collective Intelligence in Embodied AI](https://mlsanity.com/paper/2609.11737/) (2026)
- [Rethinking Scientific Modeling: Toward Physically Consistent and Simulation-Executable Programmatic Generation](https://mlsanity.com/paper/2602.07083/) (2026)
- [Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognition](https://mlsanity.com/paper/2603.17558/) (2026)
- [WARBENCH: A Comprehensive Benchmark for Evaluating LLMs in Military Decision-Making](https://mlsanity.com/paper/2603.21280/) (2026)
- [SHIELD: A Diverse Clinical Note Dataset and Distilled Small Language Models for Enterprise-Scale De-identification](https://mlsanity.com/paper/2605.03301/) (2026)
- [Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion](https://mlsanity.com/paper/2605.11679/) (2026)
- [Do Vision-Language-Action Models Mean What They Say? On the Role of Faithfulness in Embodied Reasoning](https://mlsanity.com/paper/2607.04681/) (2026)
- [Maximum Likelihood Estimation for Entity Ranking under Iterative Synthetic Data Augmentation](https://mlsanity.com/paper/2609.32987/) (2026)
- [MPCEval: A Benchmark for Multi-Party Conversation Generation](https://mlsanity.com/paper/2603.04969/) (2026)
- [Domain Adapted Large Language Models for Additive Manufacturing](https://mlsanity.com/paper/2603.22017/) (2026)
- [GazeVaLM: A Multi-Observer Eye-Tracking Benchmark for Evaluating Clinical Realism in AI-Generated X-Rays](https://mlsanity.com/paper/2604.11653/) (2026)
- [TIDE-Bench: Task-Aware and Diagnostic Evaluation of Tool-Integrated Reasoning](https://mlsanity.com/paper/2605.09544/) (2026)
- [Tracing the ongoing emergence of human-like reasoning in Large Language Models](https://mlsanity.com/paper/2605.21299/) (2026)
- [OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields](https://mlsanity.com/paper/2605.29833/) (2026)
- [Saliency-Aware Model Merging](https://mlsanity.com/paper/2606.00511/) (2026)
- [EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries](https://mlsanity.com/paper/2606.15735/) (2026)
- [NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research](https://mlsanity.com/paper/2606.26671/) (2026)
- [Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices](https://mlsanity.com/paper/2607.10183/) (2026)
- [Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations](https://mlsanity.com/paper/2608.00711/) (2026)
- [Physico-Geospatial Grounded Scene Interpretation for Mobile Robotics](https://mlsanity.com/paper/2609.06629/) (2026)
- [Syndrome, Synergy, and Safety: Structured Reasoning and Knowledge-Driven Alignment for TCM Prescription Generation](https://mlsanity.com/paper/2609.25755/) (2026)
- [One Model Is Not a Crowd: Multi-LLM and Aspect-Conditioned Diverse Comment Generation](https://mlsanity.com/paper/2609.33666/) (2026)
- [PowerBench: A Benchmark for Agentic Retrieval and Reasoning in Power Systems](https://mlsanity.com/paper/2609.34492/) (2026)

## Abstract (arXiv)

This document consolidates publicly reported technical details about Metas Llama 4 model family. It summarizes (i) released variants (Scout and Maverick) and the broader herd context including the previewed Behemoth teacher model, (ii) architectural characteristics beyond a high-level MoE description covering routed/shared-expert structure, early-fusion multimodality, and long-context design elements reported for Scout (iRoPE and length generalization strategies), (iii) training disclosures spanning pre-training, mid-training for long-context extension, and post-training methodology (lightweight SFT, online RL, and lightweight DPO) as described in release materials, (iv) developer-reported benchmark results for both base and instruction-tuned checkpoints, and (v) practical deployment constraints observed across major serving environments, including provider-specific context limits and quantization packaging. The manuscript also summarizes licensing obligations relevant to redistribution and derivative naming, and reviews publicly described safeguards and evaluation practices. The goal is to provide a compact technical reference for researchers and practitioners who need precise, source-backed facts about Llama 4.