mlsanityData to 14 Oct 2026

Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models

arXiv 2401.17139 ↗First posted 30 Jan 2024cs.LG cs.AI cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have transformed natural language processing and extended their powerful capabilities to multi-modal domains. As LLMs continue to advance, it is crucial to develop diverse and appropriate metrics for their evaluation. In this paper, we introduce a novel rank-based metric, Diff-eRank, grounded in information theory and geometry principles. Diff-eRank assesses LLMs by analyzing their hidden representations, providing a quantitative measure of how efficiently they eliminate redundant information during training. We demonstrate the applicability of Diff-eRank in both single-modal (e.g., language) and multi-modal settings. For language models, our results show that Diff-eRank increases with model size and correlates well with conventional metrics such as loss and accuracy. In the multi-modal context, we propose an alignment evaluation method based on the eRank, and verify that contemporary multi-modal LLMs exhibit strong alignment performance based on our method. Our code is publicly available at https://github.com/waltonfuture/Diff-eRank.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

31 of 48 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
19 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  4. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  5. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  7. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  8. 2020Evaluation of Text Generation: A Surveycited by 277
  9. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  10. 2023Language Modeling Is Compressioncited by 225
  11. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  12. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  13. 2021Conditional probing: measuring usable information beyond a baselinecited by 62
  14. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
  15. 2023LLMZip: Lossless Text Compression using Large Language Modelscited by 44
  16. 2023DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropiescited by 21
  17. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  18. 2023Information Flow in Self-Supervised Learningcited by 16
  19. 2023RelationMatch: Matching In-batch Relationships for Semi-supervised Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 16; 2026: 12.

35 papers.

Sort:

By area:

  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  2. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 50
  3. 2025Advancing Multimodal Reasoning via Reinforcement Learning with Cold Startcited by 41
  4. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  5. 2024Rethinking Data Selection at Scale: Random Selection is Almost All You Needcited by 22
  6. 2024Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Ratecited by 20
  7. 2025Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughtscited by 19
  8. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 16
  9. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  10. 2025Layer-wise dynamic rank for compressing large language modelscited by 8
  11. 2024Can I understand what I create? Self-Knowledge Evaluation of Large Language Modelscited by 6
  12. 2025Entropy-Lens: Uncovering Decision Strategies in LLMscited by 6
23 more
  1. 2025Enhancing Cross-task Transfer of Large Language Models via Activation Steeringcited by 6
  2. 2024The Information of Large Language Model Geometrycited by 5
  3. 2024Lightweight Safety Classification Using Pruned Language Modelscited by 5
  4. 2025Visual Reasoning Evaluation of Grok, Deepseek Janus, Gemini, Qwen, Mistral, and ChatGPTcited by 5
  5. 2026Large Vision-Language Models Get Lost in Attentioncited by 5
  6. 2025Geometric Metrics and LLMs: What They Measure and When They Workcited by 4
  7. 2024Unveiling the Dynamics of Information Interplay in Supervised Learningcited by 3
  8. 2026NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networkscited by 3
  9. 2025Beyond External Monitors: Enhancing Transparency of Large Language Models for Easier Monitoringcited by 2
  10. 2025Timber: Training-free Instruct Model Refining with Base via Effective Rankcited by 2
  11. 2025Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?cited by 2
  12. 2026Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Lawscited by 2
  13. 2025Relative-Based Scaling Law for Neural Language Modelscited by 1
  14. 2026Rank Is Not Capacity: Spectral Occupancy for Latent Graph Modelscited by 1
  15. 2026Anytime-Valid LLM Leaderboards via Benchmark-weighted and Block-Factorized e-Processescited by 1
  16. 2024Combining Entropy and Matrix Nuclear Norm for Enhanced Evaluation of Language Modelscited by 0
  17. 2025Information-Theoretic Complementary Prompts for Improved Continual Text Classificationcited by 0
  18. 2026Efficient, Validation-Free Intrinsic Quality Estimation for Large-Scale Face Recognition Datasetscited by 0
  19. 2026Latent Performance Profiling of Large Language Modelscited by 0
  20. 2026RankVR: Low-Rank Structure Perception and Value Recalibration for Robust Composed Image Retrievalcited by 0
  21. 2026SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendationcited by 0
  22. 2026Seeing Before Answering: Training-Free Visual Layer Profiling for Vision-Language Modelscited by 0
  23. 2026Do Large Language Models Capture the Diversity in their Training Data?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages
  2. 2025Multi-Scale Manifold Alignment for Interpreting Large Language Models: A Unified Information-Geometric Framework
  3. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMs
  4. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillation
  5. 2025A Survey on Training-free Alignment of Large Language Models
  6. 2026Geometric Properties of the Voronoi Tessellation in Latent Semantic Manifolds of Large Language Models
  7. 2024Efficient Alignment of Large Language Models via Data Sampling
  8. 2023A Comprehensive Overview of Large Language Models