mlsanityData to 14 Oct 2026

Benchmarking quantized LLaMa-based models on the Brazilian Secondary School Exam

arXiv 2309.12071 ↗First posted 21 Sep 2023cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Although Large Language Models (LLMs) represent a revolution in the way we interact with computers, allowing the construction of complex questions and the ability to reason over a sequence of statements, their use is restricted due to the need for dedicated hardware for execution. In this study, we evaluate the performance of LLMs based on the 7 and 13 billion LLaMA models, subjected to a quantization process and run on home hardware. The models considered were Alpaca, Koala, and Vicuna. To evaluate the effectiveness of these models, we developed a database containing 1,006 questions from the ENEM (Brazilian National Secondary School Exam). Our analysis revealed that the best performing models achieved an accuracy of approximately 46% for the original texts of the Portuguese questions and 49% on their English translations. In addition, we evaluated the computational efficiency of the models by measuring the time required for execution. On average, the 7 and 13 billion LLMs took approximately 20 and 50 seconds, respectively, to process the queries on a machine equipped with an AMD Ryzen 5 3600x processor

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 17 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

14 of 17 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  9. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  10. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  11. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  12. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
2 more
  1. 2019Word Embeddings: A Surveycited by 45
  2. 2023Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Examscited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct
  2. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
  3. 2024Quantifying the Capabilities of LLMs across Scale and Precision
  4. 2026BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams
  5. 2026mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?
  6. 2024Porting Large Language Models to Mobile Devices for Question Answering
  7. 2026The nextAI Solution to the NeurIPS 2023 LLM Efficiency Challenge
  8. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective