#521: Red Teaming LLMs and GenAI with PyRIT
67
Companies
100
Key Quotes
3
Topics
5
Insights
🎯 Summary
Podcast Summary: #521: Red Teaming LLMs and GenAI with PyRIT
This episode of Talk Python to Me features Tory Westerhof (Operations Lead for Microsoft’s AI Red Team) and Roman Lutz (Engineer on the AI Red Team tooling side) discussing the emerging field of AI security, specifically focusing on red teaming Large Language Models (LLMs) and Generative AI (GenAI) systems. The core of the discussion revolves around identifying new attack surfaces created when LLMs are connected to tools and external data, and introducing PyRIT (Pythonic Red Teaming), an open-source framework developed by the Microsoft AI Red Team to automate and scale security testing.
1. Focus Area
The primary focus is AI/ML Security and Red Teaming for LLMs and Agentic Systems. Key technologies discussed include LLMs, prompt engineering/injection, agentic AI architectures, and the application of traditional security methodologies (like OWASP) to this new domain. The central tool discussed is PyRIT.
2. Key Technical Insights
- The New Attack Surface: Connecting LLMs to tools, APIs, and untrusted external data (like documents or emails) creates a fundamentally new and complex attack surface, moving beyond traditional application security boundaries.
- Indirect Prompt Injection as the Evolving Threat: While direct prompt injection is the “bread and butter,” indirect prompt injection (where malicious instructions are hidden in ingested data, like an email or spreadsheet) is seen as the most rapidly evolving and difficult-to-test vector as AI systems become more integrated (agentic).
- PyRIT’s Role in Scaling Security: PyRIT is designed to automate the tedious aspects of red teaming, allowing security teams to test defenses against a vast and evolving set of adversarial prompts and scenarios, turning security from a one-time audit into an everyday engineering practice.
3. Business/Investment Angle
- Mismatch in AI Project Success: The high reported failure rate of AI projects (cited at 95%) might stem from unrealistic expectations (e.g., trying to replace entire call centers) rather than failures in internal developer tooling, where agentic AI is already proving highly beneficial for productivity.
- Security as a Prerequisite for Adoption: For broader, user-facing AI adoption to succeed, robust safety and security measures must be prioritized. Security is the necessary foundation for realizing the benefits of agentic systems.
- The Value of Security-Minded Talent: The speakers humorously noted that candidates who demonstrate clever indirect prompt injection techniques in their resumes might actually be highly desirable for their adversarial thinking, highlighting the need for security-aware engineering talent in this space.
4. Notable Companies/People
- Microsoft AI Red Team: The organization responsible for pioneering these testing methodologies and developing PyRIT.
- Tory Westerhof & Roman Lutz: Guests and key contributors to the Microsoft AI Red Team, focusing on testing high-risk GenAI systems and building the necessary tooling, respectively.
- OWASP: Mentioned for establishing the OWASP Top 10 for LLM Applications and Generative AI Vulnerabilities, providing a foundational framework for understanding risks like prompt injection.
5. Future Implications
The industry is rapidly moving toward highly integrated, agentic systems that interact with complex data sources and tools. This necessitates a shift in security thinking:
- Continuous Testing: Security must be integrated throughout the development lifecycle (Shift Left), moving away from post-deployment audits.
- Interdisciplinary Expertise: Effective red teaming requires input from diverse fields, including traditional cybersecurity, law, and domain expertise, to model complex harms.
- Defining Agency Boundaries: Future work will focus heavily on taxonomy and control mechanisms to prevent “excessive agency”—where agents operate without sufficient human-in-the-loop oversight or possess autonomous capabilities deemed too high-risk.
6. Target Audience
This episode is highly valuable for AI/ML Engineers, Security Professionals (DevSecOps, Red Teamers), Software Architects, and Technology Leaders responsible for deploying LLM-powered applications into production environments.
🏢 Companies Mentioned
GitHub
✅
technology_tool
Red Hat
✅
technology_partner
Oracle
✅
technology_partner
Google Cloud
✅
big_tech
Dell Technologies
✅
technology_partner
Cisco
✅
technology_partner
DeepSeek
✅
ai_company
MIT
✅
ai_research
That Python
✅
unknown
So Pirate
✅
unknown
What Pirate
✅
unknown
Azure OpenAI
✅
unknown
Azure Data Center
✅
unknown
Red Teamer
✅
unknown
AI Red Teaming
✅
unknown
💬 Key Insights
"Python is the language where all the research comes out, just because it's fairly high-level and people in the research community love to not have to write so many parentheses and brackets and things, I guess. So it's not as verbose as some other languages. You can get things done fast."
"Arguably, one of the hardest problems in all of Pirate and this entire AI red teaming space with automation is getting the scoring right."
"The shortest description I can give you about Pirate is that we're using adversarial LLMs to attack other LLMs, and yet another LLM decides whether it worked or not, and then you iterate on that."
"We're thinking about agents where we do not have insight or the correct human-in-the-loop controls on performance or execution of action."
"And we also have a world where models themselves have autonomous control capabilities. And that means the model itself, irrespective of the system that it's integrated into, has autonomous capabilities that we would deem high capabilities, right? Kind of in examples of the last bit could be self-replication of a model or self-editing of a model."
"We think about it in two ways. There's traditional security vulnerabilities with agents and agency generally, but we're thinking about the excessive agency element. We're thinking about agents where we do not have insight or the correct human-in-the-loop controls on performance or execution of action."
📊 Topics
#artificialintelligence
143
#generativeai
9
#aiinfrastructure
1
🧠 Key Takeaways
be doing this episode in the Caribbean, don't you? We should
have talked to Microsoft and say, look, I don't believe we're going to be able to do a proper representation of this unless we're on a beach in Jamaica or somewhere, but here we are
definitely at least talk to them because they're on the right track