Prompt injection is the new phishing — assume it, contain it, rehearse the crisis
AI security, resilience, and crisis management
The question: How do we make AI systems robust enough for real business use?
AI adds new attack surfaces — and agents raise the stakes because model outputs trigger real actions. Security is a board issue when your assistant reads untrusted content with your permissions. Resilience is designed, and crises are rehearsed.
What the lesson covers
The OWASP LLM Top 10 names the terrain: prompt injection (direct: the user manipulates; indirect: hostile instructions hide inside content the model reads — a web page, an email, a retrieved document), sensitive-information disclosure, data poisoning (corrupting training or knowledge sources), supply-chain risks (models, libraries, plugins), excessive agency (tools beyond need), model theft, and insecure output handling (treating model output as trusted code/commands).
Prompt injection deserves its own paragraph because it exploits the model's core competence — following instructions — and no reliable universal fix exists. Defence is layered: treat all retrieved/external content as data, never instructions (and say so in the system prompt); sanitise and provenance-filter what enters the context; validate outputs (no credentials, no unexpected actions/URLs); and cap the blast radius with least-privilege tools so a successful injection steals little and breaks less.
Whole-chain thinking: AI systems combine application, data, model, and workflow vulnerabilities. The RAG case is canonical — permissions enforced in the UI while the index retrieved everything meant any user could extract confidential strategy by asking nicely. Access control must bind at RETRIEVAL time; logs must record what was retrieved for whom.
Agents multiply stakes: outputs become tool calls, so an injected agent doesn't just say something wrong — it DOES something wrong, at machine speed. Controls from Lesson 10 become security controls: sandboxing, least privilege, action caps, human checkpoints at irreversible steps, full audit logs, kill switches. Red-teaming (attack your own system: injections, exfiltration attempts, bias probes) belongs in the release process and on a cadence, not as a one-off.
Resilience = prevention + detection + containment + recovery + COMMUNICATION. AI incidents are weird: no CVE, confused root cause, public confusion ("your bot said WHAT?"). A crisis playbook needs: severity levels, roles, the first-hour checklist (contain: kill switch; preserve: logs; assess: scope), customer and regulator communication templates (facts, impact, remediation — no speculation), and post-incident learning that feeds controls. Tabletop it before you need it.
Key points
- Know the OWASP LLM Top 10; prompt injection (especially indirect) has no universal fix — defence is layered.
- External content is data, never instructions; validate outputs; cap blast radius with least privilege.
- Permissions bind at retrieval time; log what was retrieved for whom.
- Agents turn injection into action — Lesson 10's containment controls ARE security controls.
- Rehearse the crisis: severity levels, roles, first-hour checklist, comms templates, post-incident learning.
Framework — AI Security Control Layers
data permissions → prompt boundaries → tool permissions → output validation → monitoring → incident playbooks. Six layers; assume any single layer fails and check what the next one catches. Defence-in-depth, applied to models.
The lab
Attack a design, then respond to the breach — on the clock.
Open this lesson, its lab and its quiz