mlsanityData to 7 Oct 2026

ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training

arXiv 2505.17331 ↗First posted 22 May 2025cs.LG cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

This paper introduces ECHO-LLaMA, an efficient LLaMA architecture designed to improve both the training speed and inference throughput of LLaMA architectures while maintaining its learning capacity. ECHO-LLaMA transforms LLaMA models into shared KV caching across certain layers, significantly reducing KV computational complexity while maintaining or improving language performance. Experimental results demonstrate that ECHO-LLaMA achieves up to 77% higher token-per-second throughput during training, up to 16% higher Model FLOPs Utilization (MFU), and up to 14% lower loss when trained on an equal number of tokens. Furthermore, on the 1.1B model, ECHO-LLaMA delivers approximately 7% higher test-time throughput compared to the baseline. By introducing a computationally efficient adaptation mechanism, ECHO-LLaMA offers a scalable and cost-effective solution for pretraining and finetuning large language models, enabling faster and more resource-efficient training without compromising performance.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

23 of 31 references.

References · 23

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  10. 2019Root Mean Square Layer Normalizationcited by 1,425
  11. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  12. 2020Aligning AI With Shared Human Valuescited by 706
11 more
  1. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  2. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  3. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  5. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  6. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  7. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  8. 2024Beyond KV Caching: Shared Attention for Efficient LLMscited by 10
  9. 2023GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Valuescited by 9
  10. 2023Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantizationcited by 8
  11. 2023Speeding up Resnet Architecture with Layers Targeted Low Rank Decompositioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inferencecited by 1
  2. 2026EPAS: Efficient Training with Progressive Activation Sharingcited by 0
  3. 2026KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMscited by 0