This workshop, first presented at Black Hat USA 2024, empowers security professionals to systematically red team AI systems. It goes beyond traditional security failures by incorporating novel adversarial machine learning and Responsible AI (RAI) failures, enabling a holistic approach to identifying potential issues before an AI system is deployed.

The curriculum, adapted from our internal upskilling programs designed for other red teams at Microsoft, is tailored to meet the demands of safeguarding AI technologies. Participants will gain practical insights into AI architecture, functionality, and real-world applications, with a focus on popular systems like GPT, Stable Diffusion, and LLaMA.

The workshop is designed to be hands-on, providing an environment where participants can actively engage with custom-built AI applications. Practical modules challenge participants to red team these applications, while subsequent sessions offer proven strategies for fortifying them. Attendees will explore various prompting strategies for LLMs, including zero-shot, few-shot, and advanced techniques such as Retrieval-Augmented Generation.

In this training, participants will learn to red team AI systems using innovative methodologies that address emerging risks, following practices pioneered by the Microsoft AI Red Team (AIRT). The course covers AI-specific vulnerabilities, Responsible AI (RAI) vulnerabilities, and effective mitigations. Participants will engage in a comprehensive learning experience through a blend of lectures, hands-on playground sessions, and coding labs.

By the end of the course, attendees will be equipped to probe any machine learning system for vulnerabilities, including prompt injection attacks, using both manual and automated methods. The coding portion will introduce participants to PyRIT, AIRT’s open-source toolkit for red teaming. Additionally, since the course is hosted by Microsoft, attendees will have the unique opportunity to examine real vulnerabilities discovered in Microsoft products and services, illustrating the “break-fix” cycle that the AI Red Team employs.

This course is led by experts from Microsoft’s AI Red Team, the first team to integrate RAI red teaming alongside traditional security red teaming. Over the past year, this team has assessed every high-risk AI system at Microsoft, including foundation models and Copilots.

Attendees will need to bring a computer with internet access and accounts and API keys for 

2 Days

AI Red Teaming in Practice

Application Security

Hacking

Process

Security Tooling

Using this foundational knowledge, we will move into understanding responsible AI (RAI) practices, covering topics like fairness (bias evaluation and mitigation), explainability, and privacy. Attendees will explore actionable strategies to incorporate RAI principles into their workflows, ensuring models are not only functional but also ethical.

Throughout this workshop, we will use hands-on exercises using Python-based tools and provided datasets, with options to work either on personal laptops or hosted environments like Google Colab.

Machine Learning Foundations for Developers

Machine Learning

Many developers, data scientists and companies are rushing to adopt and leverage these developments in their business process, products and solutions.

What do I need to consider when developing an LLM based solution?

What is a generative pre-trained transformer, and how does it work?

How can these models be leveraged by application developers to deliver real business value?

In this workshop, you will explore how to leverage the OpenAI and 3rd party models to develop AI enhanced solutions. You will learn options for hosting and consuming different models and develop code and prompts to explore their capabilities. The integration with company knowledgebases to provide a ChatGPT experience tailored to your business requirements will be covered along with the concepts of using prompt engineering to influence the outputs generated by the model.

Hands-on labs using the latest LLM frameworks will be available in C# and Python, ranging from walk-through exercises to advanced challenges and group activities.

Open AI & GPT Workshop

.NET

The course blends actionable guidance (e.g., AI-assisted coding security) with hands-on insights into securing AI itself (e.g., protecting model endpoints, managing drift, and red teaming AI pipelines). After completing this workshop, participants will be equipped with both the technical know-how and the processes needed to embed security into every stage of their AI-driven products.

Day 1: Using AI to Build Secure Software


Key AI security concepts and common threats

Overview of AI-related vulnerabilities in software development

Quick review of how attackers exploit AI-driven software

Understand the foundational principles of AI security

Recognize the broad threat landscape surrounding AI

Using AI-powered tools (e.g., ChatGPT, Copilot) for code generation

Common pitfalls and vulnerabilities (injected flaws, licensing, dependency issues)

Assess the security implications of AI-driven code generation

Adopt best practices to minimize vulnerabilities introduced by auto-generated code

Integrating security throughout AI/ML system development (planning, data ingestion, modeling, deployment)

Secure DevOps and MLOps practices (CI/CD with security gates)

Handling sensitive data used to train AI models

Incorporate security gates into each phase of AI software development

Properly manage and protect training data and models

Unique threat modeling considerations for AI/ML pipelines (data poisoning, model extraction)

Differences between typical web-app threat modeling vs. AI-centric threats

Identify critical assets and attack vectors in AI workflows

Apply threat modeling to preempt attacks unique to AI pipelines

5. OWASP Top 10 for Large Language Model (LLM) Applications

Top vulnerabilities specific to LLM-based applications

Prompt injection, data exfiltration, malicious chat manipulation

Recommendations, mitigations, and real-world case studies

Recognize and remediate LLM-specific security flaws

Secure LLM-based features in a production environment

Third-party libraries, datasets, and pre-trained models

Checking trustworthiness of external dependencies (data lineage, library patch levels)

Understand how AI supply chain vulnerabilities can undermine entire systems

Implement best practices to vet and protect external model/data sources

Types of adversarial attacks (evasion, poisoning, model inversion)

Techniques to detect and mitigate adversarial inputs

Gradient masking pitfalls and robust model training

Recognize adversarial attacks and how they manifest

Integrate detection and defense strategies into AI models

How to structure AI-specific penetration tests

Tools and frameworks for adversarial testing

Red vs. Blue Team exercises for AI pipelines

Conduct targeted security assessments on AI systems

Operationalize red teaming to continuously improve defenses

Lifecycle management for AI models in production

Security-driven versioning and rollback strategies

Responding quickly to emerging threats or discovered vulnerabilities

Implement robust model update/patching workflows

Reduce downtime and exposure during patch cycles

5. AI Model Interpretability and Security

Why interpretability is critical for detecting malicious behavior or biases

How more interpretable models can reveal vulnerabilities (model extraction, inversion attacks)

Balance interpretability with security concerns

Use interpretability tools to spot anomalies and potential threats

Securing AI workloads and vector databases with strict role-based and zero-trust principles

Credential management for AI model endpoints (API tokens, ephemeral credentials)

Architect AI systems with principle-of-least-privilege at every layer

Apply zero-trust approaches to protect critical AI resources

7. AI Model Drift and Security Monitoring

How models degrade over time (concept drift, data distribution shift)

Continuous monitoring and anomaly detection in production

Mitigating risks introduced by drift (potential new vulnerabilities)

Set up effective monitoring to detect model performance and security anomalies

Know when/how to trigger model retraining or security re-evaluations

NDC { AI }

Workshops

09:00 - 17:00 (UTC+01)

Open AI & GPT Workshop

Alan Smith

Building and Deploying Secure AI: Practical Strategies for Developers

Jim Manico

Machine Learning Foundations for Developers

Michelle Frost

AI Red Teaming in Practice

Gary Lopez