mlsanityData to 7 Oct 2026

KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks

arXiv 2410.06526 ↗First posted 9 Oct 2024cs.DBcited by 67

Not on the map. Why?

Cited by 28 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we introduce Knowledge-Orthogonal Reasoning (KOR), a concept aimed at minimizing reliance on domain-specific knowledge, enabling more accurate evaluation of models' reasoning abilities in out-of-distribution settings. Based on this concept, we propose the Knowledge-Orthogonal Reasoning Benchmark (KOR-Bench), encompassing five task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. KOR-Bench emphasizes models' effectiveness in applying new rule descriptions to solve novel rule-driven questions. O1-Preview and O1-Mini achieve accuracies of 72.88% and 70.16%, surpassing Claude-3.5-Sonnet and GPT-4o (58.96% and 58.00%), highlighting the effectiveness of KOR-Bench. We perform detailed analyses, identifying bottlenecks in the Cipher task with Stepwise Prompting, where two rounds of Self-Correction yield optimal results. We evaluate performance across three integrated tasks, explore the impact of Tricks on the Puzzle task, and visualize rule-focused attention. Additionally, we conduct an ablation study on dataset size, benchmark correlations, and zero-shot and three-shot "only questions" experiments. KOR-Bench aims to enhance reasoning evaluation and support further research in this area.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Oct 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 40; 2026: 27.

67 papers.

Sort:

By area:

  1. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  2. 2025AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learningcited by 86
  3. 2025CodeI/O: Condensing Reasoning Patterns via Code Input-Output Predictioncited by 54
  4. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  5. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 53
  6. 2026The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligencecited by 50
  7. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 48
  8. 2025SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyondcited by 47
  9. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  10. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 39
  11. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  12. 2026LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routingcited by 35
38 more
  1. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
  2. 2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 26
  3. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimizationcited by 21
  4. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 21
  5. 2025InternBootcamp: Boosting LLM Reasoning with Verifiable Task Scalingcited by 19
  6. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  7. 2025CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesiscited by 15
  8. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  9. 2025OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Valuecited by 15
  10. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 15
  11. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  12. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  13. 2025Reinforcement Learning on Pre-Training Datacited by 12
  14. 2025CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challengescited by 11
  15. 2025CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Creditcited by 11
  16. 2025NP-Engine: Empowering Optimization Reasoning in Large Language Models with Verifiable Synthetic NP Problemscited by 11
  17. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  18. 2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoningcited by 9
  19. 2026Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrowcited by 8
  20. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  21. 2025MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?cited by 7
  22. 2025Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challengescited by 7
  23. 2025KORGym: A Dynamic Game Platform for LLM Reasoning Evaluationcited by 7
  24. 2026MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Mergingcited by 7
  25. 2025SLR: Automated Synthesis for Scalable Logical Reasoningcited by 6
  26. 2025IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewardscited by 6
  27. 2025Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusionscited by 6
  28. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Taskscited by 6
  29. 2026Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scalecited by 6
  30. 2025CryptoX : Compositional Reasoning Evaluation of Large Language Modelscited by 5
  31. 2025Contrastive Representations for Temporal Reasoningcited by 5
  32. 2026ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Modelscited by 5
  33. 2025BOSE: A Systematic Evaluation Method Optimized for Base Modelscited by 4
  34. 2025DONOD: Efficient and Generalizable Instruction Fine-Tuning for LLMs via Model-Intrinsic Dataset Pruningcited by 4
  35. 2025Reasoning Can Hurt the Inductive Abilities of Large Language Modelscited by 4
  36. 2025Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasetscited by 4
  37. 2026Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verificationcited by 4
  38. 2026When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoningcited by 4

50 most cited of 67 All 67 (JSON)