mlsanityData to 14 Oct 2026

Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development

arXiv 2508.10108 ↗First posted 13 Aug 2025cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

AI systems for software development are rapidly gaining prominence, yet significant challenges remain in ensuring their safety. To address this, Amazon launched the Trusted AI track of the Amazon Nova AI Challenge, a global competition among 10 university teams to drive advances in secure AI. In the challenge, five teams focus on developing automated red teaming bots, while the other five create safe AI assistants. This challenge provides teams with a unique platform to evaluate automated red-teaming and safety alignment methods through head-to-head adversarial tournaments where red teams have multi-turn conversations with the competing AI coding assistants to test their safety alignment. Along with this, the challenge provides teams with a feed of high quality annotated data to fuel iterative improvement. Throughout the challenge, teams developed state-of-the-art techniques, introducing novel approaches in reasoning-based safety alignment, robust model guardrails, multi-turn jail-breaking, and efficient probing of large language models (LLMs). To support these efforts, the Amazon Nova AI Challenge team made substantial scientific and engineering investments, including building a custom baseline coding specialist model for the challenge from scratch, developing a tournament orchestration service, and creating an evaluation harness. This paper outlines the advancements made by university teams and the Amazon Nova AI Challenge team in addressing the safety challenges of AI for software development, highlighting this collaborative effort to raise the bar for AI safety.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 63 references that are in the map. First posted on arXiv 13 Aug 2025.

Builds on

34 of 63 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
22 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  6. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  7. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  8. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  9. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
  10. 2023How Secure is Code Generated by ChatGPT?cited by 81
  11. 2024Instruction Tuning for Secure Code Generationcited by 68
  12. 2023LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluationscited by 62
  13. 2025The Amazon Nova Family of Models: Technical Report and Model Cardcited by 41
  14. 2023Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attackscited by 33
  15. 2018Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prizecited by 29
  16. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 28
  17. 2025PurpCode: Reasoning for Safer Code Generationcited by 18
  18. 2024Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answerscited by 13
  19. 2023MITRE ATT&CK: State of the Art and Way Forwardcited by 12
  20. 2025RedCoder: Automated Multi-Turn Red Teaming for Code LLMscited by 9
  21. 2025ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistantscited by 6
  22. 2023Alexa, play with robot: Introducing the First Alexa Prize SimBot Challenge on Embodied AIcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Lessons From Red Teaming 100 Generative AI Products
  2. 2025OpenAI's Approach to External Red Teaming for AI Models and Systems
  3. 2024Standardization Trends on Safety and Trustworthiness Technology for Advanced AI
  4. 2025Evaluating the Critical Risks of Amazon's Nova Premier under the Frontier Model Safety Framework
  5. 2025Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition
  6. 2023ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models
  7. 2020An overview of 11 proposals for building safe advanced AI
  8. 2025A Red Teaming Roadmap Towards System-Level Safety