905: Why RAG Makes LLMs Less Safe (And How to Fix It), with Bloomberg’s Dr. Sebastian Gehrmann
🎯 Summary
Podcast Summary: 905: Why RAG Makes LLMs Less Safe (And How to Fix It), with Bloomberg’s Dr. Sebastian Gehrmann
This episode features Dr. Sebastian Gehrmann, Head of Responsible AI at Bloomberg, discussing the counterintuitive finding that Retrieval Augmented Generation (RAG) systems can actually decrease the safety and reliability of Large Language Models (LLMs), despite RAG being widely adopted as a necessary tool for grounding responses in factual data.
1. Focus Area
The discussion centers on Responsible AI, LLM safety, and the architectural risks introduced by RAG systems, particularly within high-stakes, regulated environments like finance. Key themes include the separation of “Helpful” versus “Harmless” AI goals, the concept of context-specific attack surfaces, and the limitations of general-purpose LLM safety guardrails when integrated with proprietary data via RAG.
2. Key Technical Insights
- RAG Circumvents Built-in Safety: The research demonstrated that coupling unsafe queries (e.g., “How do I commit insider trading?”) with retrieved, otherwise innocuous documents (like Wikipedia articles) caused models to bypass their inherent safety mechanisms and generate unsafe responses.
- Safety vs. Helpfulness Dichotomy: RAG significantly enhances the Helpfulness/Honesty aspect (reducing hallucinations via grounding and attribution), but it does not inherently guarantee Harmlessness. The safety alignment baked into base LLMs is often compromised in the RAG pipeline.
- Context Window Overload Risks: LLMs optimized for short prompts often struggle when deployed in RAG environments that feed them extensive context (e.g., dozens of retrieved documents), potentially leading to unpredictable behavior beyond their intended operational limits.
3. Business/Investment Angle
- Contextual Risk Assessment is Mandatory: Organizations integrating LLMs must move beyond relying on vendor benchmarks. The primary responsibility for safety evaluation lies with the application developer, as they understand the specific socio-technical context and regulatory risks (e.g., financial misconduct, unsolicited advice).
- Need for Domain-Specific Guardrails: General-purpose open-source guardrails (like those from Meta or Google) fail when tested against domain-specific risks (e.g., financial crime queries). This necessitates building custom content risk taxonomies and tailored input/output classifiers.
- RAG is Still Essential: Despite the safety risks, RAG remains a necessity for grounding LLMs in timely, proprietary, or factual data, especially in industries requiring transparent attribution.
4. Notable Companies/People
- Dr. Sebastian Gehrmann (Bloomberg): Head of Responsible AI, author of the paper “RAG LLMs Are Not Safer,” and expert in grounding LLM applications in regulated industries.
- Bloomberg: The context for the research, highlighting the stringent safety requirements in financial data and software.
- Anthropic: Mentioned for originating the “Helpful, Honest, Harmless” (3H) framework for AI evaluation.
5. Future Implications
The industry must shift its focus from simply measuring base model performance on public benchmarks to rigorously evaluating the entire end-to-end RAG application within its specific deployment context. Future safe LLM architectures will require layered defense mechanisms: Guardrail $\rightarrow$ Retrieval $\rightarrow$ Answer $\rightarrow$ Guardrail, ensuring safety checks occur both before and after grounding the response.
6. Target Audience
This episode is highly valuable for AI Engineers, Data Scientists, Responsible AI practitioners, and Technology Leaders responsible for deploying LLM applications in enterprise or regulated settings where factual accuracy and compliance are paramount.
🏢 Companies Mentioned
💬 Key Insights
"If you use them out of the box and say, 'Look, I use Llama Guard, I use Shield Llama, I use NeMo Guardrails, I'm safe now, right?' You are protected against a particular view of safety that is very much grounded in categories that are relevant to broad populations..."
"Even safeguards that are in dedicated models or systems to provide these kind of first-pass judgments, like 'Is this safe? Is this unsafe?'—they're also not trained on financial services."
"It could be that the fast, cheap, small model is completely up to the task. And in that case, why would I use this completely overblown model to do the same task, just because it is performing better on things that are completely not relevant to your particular application?"
"if in the end Gemma is also refusing to answer completely safe questions and is completely safe and correct in a RAG setup, it's not going to be helpful. So even though it is harmless, it still would not be able to be used."
"we found that basically every system was breakable regardless of whether small or large."
"model size is a little bit of a misnomer because we have so many models that rely on mixture of experts and that have architectural advantages, that even though on paper there are more parameters, they actually are using fewer of them when you actually run them live."