mlsanityData to 7 Oct 2026

Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layers

arXiv 2312.04333 ↗First posted 7 Dec 2023cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

This paper presents an in-depth analysis of Large Language Models (LLMs), focusing on LLaMA, a prominent open-source foundational model in natural language processing. Instead of assessing LLaMA through its generative output, we design multiple-choice tasks to probe its intrinsic understanding in high-order tasks such as reasoning and computation. We examine the model horizontally, comparing different sizes, and vertically, assessing different layers. We unveil several key and uncommon findings based on the designed probing tasks: (1) Horizontally, enlarging model sizes almost could not automatically impart additional knowledge or computational prowess. Instead, it can enhance reasoning abilities, especially in math problem solving, and helps reduce hallucinations, but only beyond certain size thresholds; (2) In vertical analysis, the lower layers of LLaMA lack substantial arithmetic and factual knowledge, showcasing logical thinking, multilingual and recognitive abilities, with top layers housing most computational power and real-world knowledge.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 20 of its 26 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

20 of 26 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
8 more
  1. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  2. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  3. 2019Assessing BERT's Syntactic Abilitiescited by 191
  4. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  5. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  6. 2023Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observationscited by 58
  7. 2023Alleviating Over-smoothing for Unsupervised Sentence Representationcited by 16
  8. 2022Bridging the Gap between Language Models and Cross-Lingual Sequence Labelingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 2.

20 papers.

Sort:

By area:

  1. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  2. 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoderscited by 43
  3. 2024Reference-free Hallucination Detection for Large Vision-Language Modelscited by 23
  4. 2025BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Modelscited by 18
  5. 2025SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignmentcited by 17
  6. 2025When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasonerscited by 16
  7. 2025Probing a Vision-Language-Action Model for Symbolic States and Integration into a Cognitive Architecturecited by 14
  8. 2025LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probingcited by 14
  9. 2025HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMscited by 8
  10. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5
  11. 2024Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucinationcited by 4
  12. 2025Exploring Persona Sentiment Sensitivity in Personalized Dialogue Generationcited by 3
8 more
  1. 2025The Compositional Architecture of Regret in Large Language Modelscited by 3
  2. 2025STEP: Staged Parameter-Efficient Pre-training for Large Language Modelscited by 2
  3. 2025Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Trainingcited by 2
  4. 2025HeadHunt-VAD: Hunting Robust Anomaly-Sensitive Heads in MLLM for Tuning-Free Video Anomaly Detectioncited by 2
  5. 2025InnerThoughts: Disentangling Representations and Predictions in Large Language Modelscited by 1
  6. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoningcited by 0
  7. 2026Hallucination-aware intermediate representation edit in large vision-language modelscited by 0
  8. 2026Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
  2. 2025Large Language Models Think Too Fast To Explore Effectively
  3. 2026Understanding Large Language Models
  4. 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning
  5. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  6. 2025A Survey on Large Language Models with some Insights on their Capabilities and Limitations
  7. 2022Towards Reasoning in Large Language Models: A Survey
  8. 2025Reasoning Capabilities and Invariability of Large Language Models