mlsanityData to 7 Oct 2026

VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use

arXiv 2605.13989 ↗First posted 13 May 2026cs.CLcited by 3

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

We present VectraYX-Nano, a 41.95M-parameter decoder-only language model trained from scratch in Spanish for cybersecurity, with a Latin-American regional focus and native tool invocation via the Model Context Protocol (MCP). The model has four contributions. (i) Corpus: VectraYX-Sec-ES, a 170M-token Spanish corpus assembled by an eight-VM distributed pipeline at ~25 USD of cloud compute and split into three curriculum phases (conversational 42M, cybersecurity 118M, offensive tooling 10M). (ii) Architecture: a 42M Transformer decoder with GQA, QK-Norm, RMSNorm, SwiGLU, RoPE and z-loss, paired with a domain-balanced 16,384-token byte-fallback BPE. (iii) Curriculum with replay across the three phases yields a monotonic loss descent (9.80 -> 3.17 -> 3.00 -> 2.16); after SFT (loss 1.74) the v2 bootstrap-ablation reference attains a conversational gate of 0.775 +/- 0.043 on B5 over N=4 seeds, and a controlled Phase-2 replay sweep over 0,5,10,25,50% saturates B5 at >=25% replay. (iv) Two empirical findings, both N=4. A controlled bootstrap-corpus ablation across v2 (OpenSubs), v4 (mC4-ES), and v6 (60/25/15 OpenSubs/mC4/Wiki) exposes a loss-versus-register inversion: lower-perplexity bootstraps yield measurably worse conversational behavior (v2 > v4 > v6 on B5 at every paired seed). The B4 (tool-selection) floor of 0.000 is a corpus-density artifact, not a capacity gate: rebalancing the SFT mixture to tool-use ratio 1:21 yields VectraYX-Nano v7, the released headline configuration, reaching B4 = 0.230 +/- 0.052 at 42M while retaining B1 = 0.332 +/- 0.005 and B5 = 0.725 +/- 0.130; a LoRA replication on a 260M from-scratch mid-tier reaches 0.445 +/- 0.201. The released GGUF is 96 MB in F16, runs sub-second TTFT on commodity hardware under llama.cpp, and is, to our knowledge, the first published Spanish-native cybersecurity LLM with end-to-end MCP integration.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 28 of its 54 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

28 of 54 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
16 more
  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2019Root Mean Square Layer Normalizationcited by 1,425
  3. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  8. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  10. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  11. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  12. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  13. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  14. 2023PentestGPT: An LLM-empowered Automatic Penetration Testing Toolcited by 94
  15. 2023Spanish Pre-trained BERT Model and Evaluation Datacited by 81
  16. 2022CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domaincited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oraclecited by 1
  2. 2026VectraYX-Vision-1B: A Sub-2B Spanish/LATAM Cybersecurity Vision-Language Model with Structured Visual Reasoning and Native Tool Usecited by 0
  3. 2026Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens
  2. 2025Toward Cybersecurity-Expert Small Language Models
  3. 2026CurricuLLM: Designing Personalized and Workforce-Aligned Cybersecurity Curricula Using Fine-Tuned LLMs
  4. 2025Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques
  5. 2025DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environments
  6. 2023Advancing TTP Analysis: Harnessing the Power of Large Language Models with Retrieval Augmented Generation
  7. 2026CyberLLM-FINDS 2025: Instruction-Tuned Fine-tuning of Domain-Specific LLMs with Retrieval-Augmented Generation and Graph Integration for MITRE Evaluation
  8. 2024Crimson: Empowering Strategic Reasoning in Cybersecurity through Large Language Models