mlsanityData to 14 Oct 2026

Establishing and Evaluating Trustworthy AI: Overview and Research Challenges

arXiv 2411.09973 ↗First posted 15 Nov 2024cs.LG cs.IRcited by 12

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence (AI) technologies (re-)shape modern life, driving innovation in a wide range of sectors. However, some AI systems have yielded unexpected or undesirable outcomes or have been used in questionable manners. As a result, there has been a surge in public and academic discussions about aspects that AI systems must fulfill to be considered trustworthy. In this paper, we synthesize existing conceptualizations of trustworthy AI along six requirements: 1) human agency and oversight, 2) fairness and non-discrimination, 3) transparency and explainability, 4) robustness and accuracy, 5) privacy and security, and 6) accountability. For each one, we provide a definition, describe how it can be established and evaluated, and discuss requirement-specific research challenges. Finally, we conclude this analysis by identifying overarching research challenges across the requirements with respect to 1) interdisciplinary research, 2) conceptual clarity, 3) context-dependency, 4) dynamics in evolving systems, and 5) investigations in real-world contexts. Thus, this paper synthesizes and consolidates a wide-ranging and active discussion currently taking place in various academic sub-communities and public forums. It aims to serve as a reference for a broad audience and as a basis for future research directions.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 76 of its 204 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

76 of 204 references.

References · 76

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  5. 2016Deep Learning with Differential Privacycited by 2,728
  6. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  8. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  9. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  10. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  11. 2018Sanity Checks for Saliency Mapscited by 1,156
  12. 2020In Search of Lost Domain Generalizationcited by 1,054
38 more
  1. 2018How To Backdoor Federated Learningcited by 765
  2. 2014Certifying and removing disparate impactcited by 715
  3. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
  4. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  5. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  6. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  7. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  8. 2017On Fairness and Calibrationcited by 406
  9. 2017Methods for Interpreting and Understanding Deep Neural Networkscited by 345
  10. 2017The (Un)reliability of saliency methodscited by 333
  11. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  12. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  13. 2018On the Robustness of Interpretability Methodscited by 314
  14. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 253
  15. 2019Adversarial Attacks and Defenses in Images, Graphs and Text: A Reviewcited by 237
  16. 2018Adversarial Robustness Toolbox v1.0.0cited by 215
  17. 2018A comparative study of fairness-enhancing interventions in machine learningcited by 208
  18. 2021Trustworthy AI: From Principles to Practicescited by 207
  19. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  20. 2020Evaluating and Aggregating Feature-based Model Explanationscited by 188
  21. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 182
  22. 2022Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyondcited by 171
  23. 2018Aequitas: A Bias and Fairness Audit Toolkitcited by 147
  24. 2020Algorithmic Fairnesscited by 143
  25. 2020Synthetic Data -- Anonymisation Groundhog Daycited by 108
  26. 2017Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciencescited by 85
  27. 2020Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AIcited by 81
  28. 2021AI in Finance: Challenges, Techniques and Opportunitiescited by 81
  29. 2020On quantitative aspects of model interpretabilitycited by 71
  30. 2020Trustworthy AIcited by 62
  31. 2023Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulationcited by 61
  32. 2019Mathematical Notions vs. Human Perception of Fairness: A Descriptive Approach to Fairness for Machine Learningcited by 57
  33. 2024Design Principles for Generative AI Applicationscited by 57
  34. 2023Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystemcited by 52
  35. 2018Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failurecited by 49
  36. 2022Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvementcited by 48
  37. 2023Privacy Side Channels in Machine Learning Systemscited by 40
  38. 2021GRNN: Generative Regression Neural Network -- A Data Leakage Attack for Federated Learningcited by 36

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 1.

12 papers.

  1. 2025Formalising Human-in-the-Loop: Computational Reductions, Failure Modes, and Legal-Moral Responsibilitycited by 8
  2. 2025Generation-Time vs. Post-hoc Citation: A Holistic Evaluation of LLM Attributioncited by 6
  3. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  4. 2025A Framework for Responsible AI Systems: Building Societal Trust through Domain Definition, Trustworthy AI Design, Auditability, Accountability, and Governancecited by 5
  5. 2025Adaptive AI decision interface for autonomous electronic material discoverycited by 2
  6. 2025ATA: A Neuro-Symbolic Approach to Implement Autonomous and Trustworthy Agentscited by 2
  7. 2025I'm Sorry Dave: How the old world of personnel security can inform the new world of AI insider riskcited by 1
  8. 2025Private and Fair Machine Learning: Revisiting the Disparate Impact of Differentially Private SGDcited by 1
  9. 2025Mapping the Trust Terrain: LLMs in Software Engineering -- Insights and Perspectivescited by 0
  10. 2025Trustworthiness of Legal Considerations for the Use of LLMs in Educationcited by 0
  11. 2025PaTAS: A Framework for Trust Propagation in Neural Networks Using Subjective Logiccited by 0
  12. 2026Trustworthy AI in Digital Health: A Comprehensive Review of Robustness and Explainabilitycited by 0