# A Survey on Large Language Models with some Insights on their Capabilities and Limitations

arXiv: https://arxiv.org/abs/2501.04040 · HTML: https://mlsanity.com/paper/2501.04040/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jan 2025.

## Built on by (most cited first, 30 of 42)

- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Models](https://mlsanity.com/paper/2508.10599/) (2025)
- [MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoning](https://mlsanity.com/paper/2601.21468/) (2026)
- [LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models](https://mlsanity.com/paper/2505.19240/) (2025)
- [VL-RouterBench: A Benchmark for Vision-Language Model Routing](https://mlsanity.com/paper/2512.23562/) (2025)
- [PentestEval: Benchmarking LLM-based Penetration Testing with Modular and Stage-Level Design](https://mlsanity.com/paper/2512.14233/) (2025)
- [Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuning](https://mlsanity.com/paper/2510.10085/) (2025)
- [ALAS: A Stateful Multi-LLM Agent Framework for Disruption-Aware Planning](https://mlsanity.com/paper/2505.12501/) (2025)
- [EVADE-Bench: Multimodal Benchmark for Evaluating and Enhancing Evasive Content Detection](https://mlsanity.com/paper/2505.17654/) (2025)
- [Survey of GenAI for Automotive Software Development: From Requirements to Executable Code](https://mlsanity.com/paper/2507.15025/) (2025)
- [Large Language Model Agent for User-friendly Chemical Process Simulations](https://mlsanity.com/paper/2601.11650/) (2026)
- [KonfAI: A Modular and Fully Configurable Framework for Deep Learning in Medical Imaging](https://mlsanity.com/paper/2508.09823/) (2025)
- [ALAS: Transactional and Dynamic Multi-Agent LLM Planning](https://mlsanity.com/paper/2511.03094/) (2025)
- [What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding](https://mlsanity.com/paper/2601.09503/) (2026)
- [Agentic SPARQL: Evaluating SPARQL-MCP-powered Intelligent Agents on the Federated KGQA Benchmark](https://mlsanity.com/paper/2603.06582/) (2026)
- [In-Context Reinforcement Learning for Tool Use in Large Language Models](https://mlsanity.com/paper/2603.08068/) (2026)
- [Evaluating Ill-Defined Tasks in Large Language Models](https://mlsanity.com/paper/2603.17067/) (2026)
- [Roamify: Designing and Evaluating an LLM Based Google Chrome Extension for Personalised Itinerary Planning](https://mlsanity.com/paper/2504.10489/) (2025)
- [On Technique Identification and Threat-Actor Attribution using LLMs and Embedding Models](https://mlsanity.com/paper/2505.11547/) (2025)
- [From Transformers to Large Language Models: A systematic review of AI applications in the energy sector towards Agentic Digital Twins](https://mlsanity.com/paper/2506.06359/) (2025)
- [COMPASS: Context-Modulated PID Attention Steering System for Hallucination Mitigation](https://mlsanity.com/paper/2511.14776/) (2025)
- [Neural Probe-Based Hallucination Detection for Large Language Models](https://mlsanity.com/paper/2512.20949/) (2025)
- [NeST: Neuron Selective Tuning for LLM Safety](https://mlsanity.com/paper/2602.16835/) (2026)
- [DeltaMem: Towards Agentic Memory Management via Reinforcement Learning](https://mlsanity.com/paper/2604.01560/) (2026)
- [Edit-Based Refinement for Parallel Masked Diffusion Language Models](https://mlsanity.com/paper/2605.09603/) (2026)
- ["I know myself better, but not really greatly": How Well Can LLMs Detect and Explain LLM-Generated Texts?](https://mlsanity.com/paper/2502.12743/) (2025)
- [Movable Antenna Enhanced Federated Fine-Tuning of Large Language Models via Hybrid Client Selection Optimization](https://mlsanity.com/paper/2506.00011/) (2025)
- [InsightBuild: LLM-Powered Causal Reasoning in Smart Building Systems](https://mlsanity.com/paper/2507.08235/) (2025)
- [A Survey of the State-of-the-Art in Conversational Question Answering Systems](https://mlsanity.com/paper/2509.05716/) (2025)

## Abstract (arXiv)

The rapid advancement of artificial intelligence, particularly with the development of Large Language Models (LLMs) built on the transformer architecture, has redefined the capabilities of natural language processing. These models now exhibit remarkable performance across various language-related tasks, such as text generation, question answering, translation, and summarization, often rivaling human-like comprehension. More intriguingly, LLMs have demonstrated emergent abilities extending beyond their core functions, showing proficiency in tasks like commonsense reasoning, code generation, and arithmetic. This survey paper explores the foundational components, scaling mechanisms, and architectural strategies that drive these capabilities. Emphasizing models like GPT and LLaMA, we analyze the impact of exponential data and computational growth on LLM performance, while also addressing the trade-offs associated with scaling. We also examine LLM applications across sectors, such as healthcare, finance, education, and law, highlighting their adaptability and potential to solve domain-specific challenges. Central to this work are the questions of how LLMs generalize across diverse tasks, exhibit planning, and reasoning abilities, and whether these emergent abilities can be systematically elicited or enhanced. In particular, we provide some insights into the CoT (Chain of Thought) and PoT (Plan of Thought) abilities within LLMs, focusing on how pre-training data influences their emergence. Additionally, we investigate LLM-modulo frameworks that integrate external systems, allowing LLMs to handle complex, dynamic tasks. By analyzing these factors, this paper aims to foster the ongoing discussion on the capabilities and limits of LLMs, promoting their responsible development and application in novel and increasingly complex environments.