{"list":"other","total":61,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"2407.21783","title":"The Llama 3 Herd of Models","created":"2024-07-31","cited_by_count":14715},{"id":"2307.09288","title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","created":"2023-07-18","cited_by_count":13028},{"id":"2310.06825","title":"Mistral 7B","created":"2023-10-10","cited_by_count":3164},{"id":"2202.05262","title":"Locating and Editing Factual Associations in GPT","created":"2022-02-10","cited_by_count":2691},{"id":"2309.17453","title":"Efficient Streaming Language Models with Attention Sinks","created":"2023-09-29","cited_by_count":2186},{"id":"2304.01373","title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","created":"2023-04-03","cited_by_count":1786},{"id":"1712.05877","title":"Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference","created":"2017-12-15","cited_by_count":1593},{"id":"2211.10438","title":"SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models","created":"2022-11-18","cited_by_count":1386},{"id":"2306.00978","title":"AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration","created":"2023-06-01","cited_by_count":1298},{"id":"2211.00593","title":"Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small","created":"2022-11-01","cited_by_count":1165},{"id":"2208.07339","title":"LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale","created":"2022-08-15","cited_by_count":906},{"id":"2309.16588","title":"Vision Transformers Need Registers","created":"2023-09-28","cited_by_count":871},{"id":"2301.05217","title":"Progress measures for grokking via mechanistic interpretability","created":"2023-01-12","cited_by_count":839},{"id":"2103.13630","title":"A Survey of Quantization Methods for Efficient Neural Network Inference","created":"2021-03-25","cited_by_count":815},{"id":"2403.06764","title":"An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models","created":"2024-03-11","cited_by_count":642},{"id":"2402.00838","title":"OLMo: Accelerating the Science of Language Models","created":"2024-02-01","cited_by_count":613},{"id":"1608.04636","title":"Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-\\L{}ojasiewicz Condition","created":"2016-08-16","cited_by_count":539},{"id":"2304.14767","title":"Dissecting Recall of Factual Associations in Auto-Regressive Language Models","created":"2023-04-28","cited_by_count":528},{"id":"2206.01861","title":"ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers","created":"2022-06-04","cited_by_count":526},{"id":"2402.00159","title":"Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research","created":"2024-01-31","cited_by_count":483},{"id":"2106.08295","title":"A White Paper on Neural Network Quantization","created":"2021-06-15","cited_by_count":472},{"id":"2306.09927","title":"Trained Transformers Learn Linear Models In-Context","created":"2023-06-16","cited_by_count":337},{"id":"2310.15213","title":"Function Vectors in Large Language Models","created":"2023-10-23","cited_by_count":308},{"id":"1910.06188","title":"Q8BERT: Quantized 8Bit BERT","created":"2019-10-14","cited_by_count":300},{"id":"2402.17762","title":"Massive Activations in Large Language Models","created":"2024-02-27","cited_by_count":294},{"id":"2306.00297","title":"Transformers learn to implement preconditioned gradient descent for in-context learning","created":"2023-06-01","cited_by_count":279},{"id":"2309.14316","title":"Physics of Language Models: Part 3.1, Knowledge Storage and Extraction","created":"2023-09-25","cited_by_count":270},{"id":"2212.09720","title":"The case for 4-bit precision: k-bit Inference Scaling Laws","created":"2022-12-19","cited_by_count":251},{"id":"2205.10343","title":"Towards Understanding Grokking: An Effective Theory of Representation Learning","created":"2022-05-20","cited_by_count":230},{"id":"2410.10781","title":"When Attention Sink Emerges in Language Models: An Empirical View","created":"2024-10-14","cited_by_count":208},{"id":"2406.01721","title":"DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs","created":"2024-06-03","cited_by_count":167},{"id":"2303.06296","title":"Stabilizing Transformer Training by Preventing Attention Entropy Collapse","created":"2023-03-11","cited_by_count":161},{"id":"2306.00802","title":"Birth of a Transformer: A Memory Viewpoint","created":"2023-06-01","cited_by_count":161},{"id":"2306.12929","title":"Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing","created":"2023-06-22","cited_by_count":151},{"id":"2004.07320","title":"Training with Quantization Noise for Extreme Model Compression","created":"2020-04-15","cited_by_count":145},{"id":"2109.12948","title":"Understanding and Overcoming the Challenges of Efficient Transformer Quantization","created":"2021-09-27","cited_by_count":141},{"id":"2312.03002","title":"The mechanistic basis of data dependence and abrupt learning in an in-context classification task","created":"2023-12-03","cited_by_count":118},{"id":"2402.14735","title":"How Transformers Learn Causal Structure with Gradient Descent","created":"2024-02-22","cited_by_count":118},{"id":"2310.05249","title":"In-Context Convergence of Transformers","created":"2023-10-08","cited_by_count":115},{"id":"2310.17191","title":"How do Language Models Bind Entities in Context?","created":"2023-10-26","cited_by_count":108},{"id":"2206.04301","title":"Unveiling Transformers with LEGO: a synthetic reasoning task","created":"2022-06-09","cited_by_count":106},{"id":"2305.16380","title":"Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer","created":"2023-05-25","cited_by_count":102},{"id":"2401.12181","title":"Universal Neurons in GPT2 Language Models","created":"2024-01-22","cited_by_count":102},{"id":"2310.01082","title":"Linear attention is (maybe) all you need (to understand transformer optimization)","created":"2023-10-02","cited_by_count":98},{"id":"2310.08391","title":"How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?","created":"2023-10-12","cited_by_count":96},{"id":"2310.08566","title":"Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining","created":"2023-10-12","cited_by_count":83},{"id":"2310.10616","title":"How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations","created":"2023-10-16","cited_by_count":75},{"id":"2406.15765","title":"Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration","created":"2024-06-22","cited_by_count":74},{"id":"2406.12335","title":"Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters","created":"2024-06-18","cited_by_count":73},{"id":"2310.00535","title":"JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention","created":"2023-10-01","cited_by_count":52},{"id":"2403.01241","title":"IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact","created":"2024-03-02","cited_by_count":52},{"id":"2404.03828","title":"Outlier-Efficient Hopfield Layers for Large Transformer-Based Models","created":"2024-04-04","cited_by_count":38},{"id":"2310.12680","title":"On the Optimization and Generalization of Multi-head Attention","created":"2023-10-19","cited_by_count":37},{"id":"2408.12186","title":"Transformers are Minimax Optimal Nonparametric In-Context Learners","created":"2024-08-22","cited_by_count":37},{"id":"2009.08034","title":"Towards Fully 8-bit Integer Inference for the Transformer Model","created":"2020-09-17","cited_by_count":35},{"id":"2402.14951","title":"In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization","created":"2024-02-22","cited_by_count":33},{"id":"2406.12016","title":"Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization","created":"2024-06-17","cited_by_count":33},{"id":"2405.04669","title":"Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics","created":"2024-05-07","cited_by_count":27},{"id":"2301.12017","title":"Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases","created":"2023-01-27","cited_by_count":23},{"id":"2211.00170","title":"What is my math transformer doing? -- Three results on interpretability and generalization","created":"2022-10-31","cited_by_count":19}],"next_cursor":null}