AI Security Engineering Handbook

Most organizations know they need AI security before they know what it means. The first hire receives a mandate to own AI risk, but no one agrees on whether that means prompt injection testing, model supply chain review, governance evidence, agent authorization, or all of it at once. Every subsequent failure — the wrong hire, the shallow control, the unowned risk — usually traces back to a discipline that wasn't defined clearly enough to be operated. This chapter sets that foundation.

What This Chapter Covers

AI security engineering is the practice of protecting AI-enabled systems as engineered products, not as magic model endpoints and not as policy slogans. This chapter defines the discipline, its boundaries, and the language practitioners need when they explain the work to executives, hiring managers, software engineers, product teams, and governance stakeholders. It solves a common organizational problem: teams know AI introduces risk, but they do not know which risks belong to AppSec, ProductSec, model risk, responsible AI, GRC, platform engineering, or a new AI security function.

This chapter is relevant when an organization begins shipping LLM features, adds RAG to an existing product, gives agents access to tools, adopts third-party AI services, or starts hiring for "AI security" without knowing what the role should actually own. It is also relevant for practitioners transitioning from application security, product security, red teaming, GRC, detection engineering, or ML engineering into AI security. The career trigger is the same as the organizational trigger: familiar security instincts still matter, but the system now contains non-deterministic outputs, context as an attack surface, model artifacts, retrieval planes, eval gates, tool-call authority, and governance evidence requirements.

After working through this chapter, you should be able to explain AI security engineering in plain language, draw a boundary around the discipline, distinguish it from adjacent functions, and name the capability areas a real program must cover. You should also be able to reject weak control arguments such as "the model is responsible," "we have a policy," or "we tested some jailbreaks." Most importantly, you should be able to frame AI security differently for a CISO, a hiring manager, and a software engineer without changing the substance of the discipline.

Core Concepts

AI Security Engineering as Product Security for AI Systems AI security engineering protects systems where model behavior, context construction, retrieval, tool use, model supply chain, and AI governance evidence affect security outcomes. It inherits core AppSec and ProductSec practices: threat modeling, code review, abuse-case design, authorization, logging, secure SDLC, release gates, and incident response. It extends those practices into AI-specific surfaces such as prompt injection, context poisoning, vector-store authorization, model provenance, eval pipelines, and agent blast radius. The work is not simply "secure the model"; it is secure the system that uses the model.

The Boundary Model AI security engineering includes risks created or amplified by AI behavior inside deployed systems. In scope: LLM application security, RAG security, agent tool-calling controls, model supply chain, MLOps platform security, evals, red teaming, AI-aware SDLC, AI incident observability, vendor AI risk, privacy in AI workflows, and governance evidence. Out of scope as primary ownership: broad AI ethics strategy, abstract alignment research, general corporate compliance, ordinary cloud hardening unrelated to AI workflows, and financial model risk management unless those domains intersect with deployed AI systems. The boundary does not mean those areas are irrelevant; it means AI security engineering should not become the dumping ground for every AI concern.

Safety, Security, and Reliability Are Related but Not Identical AI safety often concerns harmful behavior, fairness, alignment, toxicity, bias, and misuse prevention. AI security concerns adversarial abuse, trust boundaries, unauthorized data access, tool misuse, supply-chain compromise, observability, and enforceable controls. Reliability concerns consistency, correctness, uptime, and performance. A hallucination may be a reliability problem, a safety problem, or a security problem depending on what property it violates and what downstream effect it creates.

Evidence Over Theater AI security engineering must produce artifacts that prove controls operated. A policy is not enough. A system prompt is not enough. A red-team report without closure evidence is not enough. Useful evidence includes threat models, eval results, release gate decisions, retrieval authorization logs, model intake records, tool-call audit trails, incident traces, risk acceptances, vendor AI reviews, and control registry entries tied to owners and cadence.

The Model Is Not a Control Argument A model can assist a control, but it cannot be the sole owner of authorization, data classification, tool permission, privacy enforcement, or release approval. A model may refuse a dangerous request, but refusal behavior is probabilistic and context-dependent. If the only thing preventing data leakage is a prompt telling the model not to leak data, the system is not secure. Durable controls live in retrieval filters, runtime authorization, schemas, tool policies, approval gates, sandboxing, logging, and release gates.

The Practitioner's Challenge

The hardest part of defining AI security engineering is that everyone arrives with a different prior model. AppSec teams see another application type. ML teams see model evaluation and training concerns. GRC teams see emerging frameworks and audit obligations. Executives see reputational and regulatory risk. Product teams see feature velocity. Each view is partially correct, but none is complete enough to run the function.

The second challenge is organizational gravity. If the discipline is defined too narrowly, it becomes "prompt injection testing" and misses retrieval, agents, model artifacts, vendor risk, and governance evidence. If it is defined too broadly, it becomes responsible for all AI risk, including ethics, legal policy, workforce change, product strategy, and broad compliance. Both failure modes are common. The first under-protects the product; the second makes the role impossible to staff or measure.

The third challenge is language. Terms such as red teaming, evals, hallucination, safety, jailbreak, model risk, and governance are used inconsistently. A practitioner who cannot disambiguate those terms will struggle to win trust. Good AI security engineers translate between groups without flattening the problem: they can tell a software engineer what to change, tell a CISO what risk remains, tell GRC what evidence exists, and tell a hiring manager what capability is missing.

How to Approach It

Start by defining the system, not the model. Ask what product workflow uses AI, what data enters, what model or provider processes it, what context is added, what tools are available, what output reaches users or systems, and what decisions depend on that output. This shifts the discussion away from abstract model behavior and toward engineered trust boundaries.

Next, classify risks by layer. The LLM application layer includes prompt assembly, output rendering, caching, streaming, and provider key handling. The retrieval layer includes authorization, metadata integrity, vector-store tenancy, and source attribution. The agent layer includes tool permissions, approvals, sandboxing, rollback, and audit logs. The supply-chain layer includes model provenance, artifact integrity, unsafe formats, and registry controls. The governance layer includes inventory, owners, evidence, and release gates.

Then define the control objective for each layer. At the application layer, the objective may be preventing boundary violations and data leakage. At the retrieval layer, it may be preventing unauthorized context assembly. At the agent layer, it may be limiting action blast radius. At the supply-chain layer, it may be proving artifact provenance and integrity. At the governance layer, it may be producing evidence that controls operate.

Use the eight capability areas as a practical capability map: AI application security, prompt and context security, RAG and data-plane security, agent and tool-use security, model supply chain security, MLOps and platform security, evals and red-team evidence, and governance-to-engineering evidence. These areas are not job titles by themselves. They are the body of work an organization must assign, staff, buy, or sequence.

Finally, practice explaining the discipline in audience-specific terms. To a CISO: "AI security engineering turns AI adoption risk into enforceable controls, evidence, and release decisions." To a hiring manager: "This role secures AI products across prompt, retrieval, model, tool, platform, and evidence surfaces; no single candidate will cover all depths equally." To a software engineer: "We are making sure the AI feature preserves authorization, data boundaries, safe tool use, logging, and rollback even when the model receives hostile or unexpected context."

Outputs and Deliverables

The core artifacts of this work start with a discipline scope statement — a document that names what AI security engineering owns, what it partners on, and what it explicitly does not own. Without it, the function expands to fill every AI concern or shrinks to whatever no one else claimed. Adjacent to the scope statement is an AI security capability map: an eight-area grid showing capability areas, example controls, likely owners, required evidence, and current maturity. Together these two documents answer the basic organizational question of what the discipline does and who does it.

The architecture work produces a boundary model diagram — a visual tracing user input through prompt orchestration, retrieval, model, tools, output path, logs, and governance artifacts, with each boundary labeled for trust level and data classification. This diagram becomes the starting point for every subsequent threat modeling session. Alongside it, a terminology guide defines hallucination, adversarial output, jailbreak, prompt injection, eval, red team, pen test, safety, security, model risk, and governance evidence in the organization's own language. Consistent vocabulary prevents confusion in incidents, hiring loops, and executive discussions, which are three very different contexts where the same words mean different things.

The operational artifacts close the set. An AI control argument template forces any feature claim through a structured question: what security property must hold, what control enforces it, where is it implemented, what evidence proves it operated, and who owns remediation. This template makes "the model will refuse" a claim that has to be defended rather than accepted. A stakeholder explanation pack translates the discipline into a CISO framing, a hiring manager framing, and an engineering framing. Not marketing polish — alignment. A discipline that cannot be explained consistently across those audiences will not be staffed or governed consistently either.

Common Failure Modes

Prompt-Injection Reductionism: The organization equates AI security with jailbreak testing. This happens because prompt attacks are visible, easy to demo, and easy for non-specialists to understand. Recover by expanding the threat model to retrieval, tools, model artifacts, MLOps, observability, privacy, and governance evidence. Keep prompt injection as a major domain, not the whole discipline.

Everything-AI Dumping Ground: The AI security role becomes responsible for all AI ethics, legal compliance, model quality, product strategy, vendor review, and security engineering at once. This happens when leaders want one owner for a complex change. Recover by defining primary ownership, partnership responsibilities, and explicit non-ownership. The function can coordinate without owning every AI concern.

Model-Centric Control Thinking: Teams assume model behavior is the main control surface. They ask the model to follow policy, refuse unsafe outputs, or avoid revealing data, while leaving retrieval, authorization, and tools weak. Avoid this by locating enforceable controls outside the model wherever possible. The model can help; it should not be the only lock.

Evidence-Free Governance: The organization writes AI policies and risk statements without connecting them to artifacts. This happens when governance moves faster than engineering implementation. Recover by mapping each governance claim to a control owner, evidence artifact, collection cadence, and release decision. If no artifact exists, the claim is not yet operational.

Implementation Checklist

• Define what AI security engineering owns, partners on, and explicitly does not own.
• Create an eight-area AI security capability map with owners, controls, and evidence.
• Draw a boundary model for at least one AI product or feature.
• Write a shared glossary for safety, security, hallucination, prompt injection, red teaming, evals, and model risk.
• Replace any "the model will refuse" control claim with an enforceable external control or documented risk acceptance.
• Identify which AI security capability areas are currently unowned in the organization.
• Prepare 30-second explanations of the discipline for a CISO, hiring manager, and software engineer.
• Tie at least one AI governance statement to a real engineering artifact and owner.

Related Reading

• Handbook chapters: Chapter 2 for role architecture and team design; Chapter 3 for threat modeling AI systems; Chapter 8 for governance-to-engineering evidence; Chapter 11 for building the operating model.
• Field Guide: AI Security Foundations for the terminology baseline and mental model boundaries; LLM Application Security for model-boundary controls; Agent Security for delegated action; AI Governance, Risk, and Compliance for evidence and ownership.

The job description that asks for an AppSec engineer, red teamer, ML engineer, governance translator, supply chain expert, privacy engineer, and security architect in one role is not hypothetical — it ships every week. It tends to attract keyword-matched candidates who can describe every AI security domain and own all of them shallowly, producing a program that generates language about risk without blocking any of it. Role design is where AI security programs work or don't.

What This Chapter Covers

AI security hiring fails when organizations compress nine distinct capability areas into one impossible job description. This chapter decomposes the Frankenstein Role into practical archetypes, staffing models, sequencing choices, and hiring language that a real security organization can use. The problem it solves is not merely recruiting inefficiency. It solves the deeper operational problem where no one knows whether the AI security hire is supposed to threat model LLM apps, red-team agents, map governance evidence, secure model registries, build evals, own RAG authorization, support model risk, or design cross-product architecture.

This chapter matters when a company writes its first AI security job description, realizes existing AppSec coverage is not enough, receives customer questions about AI controls, begins deploying RAG or agents, or watches GRC policy outpace engineering evidence. It is also relevant when a practitioner is trying to position their own career. A candidate who can explain which archetype they represent and which adjacent areas they can cover will interview better than one who claims to be expert in everything.

After working through this chapter, you should be able to split AI security work into practical archetypes, choose the right first hire by company stage, decide what to build internally versus buy or contract, and write a job description that does not inherit the Frankenstein shape. You should also be able to evaluate candidates who claim broad AI security expertise without treating breadth as automatic depth.

Core Concepts

The Frankenstein Role The Frankenstein Role appears when a job description asks one person to be an AppSec engineer, red teamer, ML engineer, governance lead, model supply-chain expert, security architect, privacy engineer, and policy translator at the same time. The role usually emerges because leadership sees AI security as one category and assumes one hire can own it. The result is a req that screens for keywords rather than capability. A better approach is to define the body of work first, then decide which archetype should own the first slice.

Archetype-Based Role Design An archetype is a practical grouping of responsibilities that commonly belong together. The nine canonical archetypes are AI Security Architect, AI Product Security Engineer, AI AppSec Engineer, RAG Security Engineer, Agent Security Engineer, AI Red Team Engineer, ML Security Engineer, Model Risk Security Partner, and Governance Evidence Lead. These are not rigid boxes; they are staffing lenses. A strong candidate may cover one archetype deeply and two adjacent areas competently, but that is different from claiming all nine.

Stage-Based Staffing A seed-stage company does not need the same AI security structure as a regulated enterprise. Early teams often need a hybrid AI AppSec/ProductSec profile who can review features, write threat models, and define release gates. Series A-B companies may need a builder plus external red-team help. Enterprises need clearer specialization, governance evidence ownership, vendor review, and architecture coordination. Regulated organizations need earlier investment in evidence, inventory, model governance, and auditability.

Build-vs-Buy Decisions Not every AI security capability needs to be staffed internally on day one. Red-team exercises, model supply-chain assessments, governance evidence mapping, and architecture reviews can be bought or contracted while the internal team builds durable ownership. Capabilities tied to daily product decisions, release gates, incident response, and internal engineering workflows usually need internal owners. Buy external depth when the need is episodic or specialized; build internal ownership when the control must operate continuously.

The Unicorn Trap The candidate who claims mastery of all AI security domains should be evaluated carefully. Broad awareness is valuable, but broad claims without artifacts often signal keyword inflation. Ask for evidence: threat models, eval suites, model intake processes, tool permission designs, governance mappings, incident traces, red-team reports, or release gates. The question is not whether the candidate has heard of every domain; it is whether they can operate at the required depth for the role you actually need.

The Practitioner's Challenge

The political challenge is that AI security often arrives after leadership has already promised AI adoption. Hiring then becomes a way to reduce anxiety: find a person who can "own AI security." That instinct is understandable, but it produces unrealistic role design. A single hire cannot simultaneously become the product reviewer, red teamer, governance translator, vendor assessor, eval engineer, and executive narrator unless the organization is willing to accept shallow coverage across most of those functions.

The structural challenge is that AI security work crosses existing boundaries. Product security owns design review, AppSec owns secure SDLC, ML platform owns training and deployment, GRC owns frameworks, privacy owns data rights, procurement owns vendors, and engineering owns product velocity. A new role that does not define interfaces with those teams will either be ignored or overloaded. Good role architecture names which decisions the AI security role owns, which it influences, and which it escalates.

The resource challenge is sequencing. Most organizations cannot hire nine archetypes immediately. They need to decide what risk is most urgent: shipping AI features safely, validating exposed systems, building governance evidence, controlling agents, securing retrieval systems, securing model artifacts, supporting model risk, or designing architecture across product lines. Hiring should follow risk and operating need, not trend language. A role built around the wrong first hire can slow the program for a year.

How to Approach It

Start with a work inventory, not a job title. List the AI systems in use, the AI features shipping soon, the data they touch, the tools they can call, the vendors involved, and the customer or regulatory pressure the organization faces. Then list the work required: threat models, reviews, red-team testing, eval gates, model intake, vendor reviews, logging, incident playbooks, evidence mapping, and hiring support.

Map that work to the nine archetypes. The AI Security Architect owns cross-cutting trust models, defense-in-depth, reference architectures, and architectural decision records. The AI Product Security Engineer owns AI feature review, product abuse paths, launch readiness, and product-team enablement. The AI AppSec Engineer owns LLM application review, prompt assembly, output handling, AI-aware secure SDLC, and developer enablement. The RAG Security Engineer owns retrieval-time authorization, source inventories, chunk metadata, tenant isolation, and retrieval test evidence. The Agent Security Engineer owns tool permissions, authorization, sandboxing, approvals, rollback, and audit trails. The AI Red Team Engineer owns adversarial testing, prompt attack libraries, eval evidence, and finding reproduction. The ML Security Engineer owns model supply chain, provenance, registries, artifact integrity, unsafe formats, and model intake. The Model Risk Security Partner owns security support for model-risk review, decision integrity, residual-risk framing, and validation evidence. The Governance Evidence Lead owns framework-to-artifact mapping, control evidence, audit readiness, and executive reporting.

Decide the first hire by operational pain. If product teams are shipping AI features without review, start with AI Product Security, AI AppSec, or AI Security Architect. If the company is already exposed and needs validation, start with red-team support or an AI Red Team Engineer. If customer assurance and audits are the burning issue, start with Governance Evidence. If RAG is central to the product, prioritize RAG Security. If agents are taking action, prioritize Agent Security. If the organization deploys many open models or fine-tunes, prioritize ML Security. If model-risk review is already a formal operating pressure, add a Model Risk Security Partner early.

Sequence stages deliberately. At seed stage, combine AI AppSec with external advisory support. At Series A-B, add repeatable SDLC and red-team capability, even if part of it is contracted. At enterprise scale, split governance evidence and architecture from hands-on product review because the volume of decisions becomes too high. In regulated environments, treat evidence and inventory as first-class early work rather than paperwork after the fact.

Write job descriptions around outcomes and artifacts. Instead of asking for "experience securing LLMs and AI/ML systems," name the deliverables: AI threat models, RAG reviews, prompt injection test plans, agent tool permission models, model intake checklists, eval gates, control evidence, and incident playbooks. This attracts candidates who have done the work and filters out candidates who only know the vocabulary.

Outputs and Deliverables

A role architecture map is the foundation. It lists the nine archetypes, each archetype's core responsibilities, adjacent areas, required artifacts, and interfaces with other teams. The map makes explicit that no single role owns every cell equally, which matters as much for setting hiring expectations as for protecting the hire from impossible scope. A company-stage staffing model sits alongside it, describing what AI security coverage looks like at seed, Series A-B, enterprise, and regulated-company stages: internal roles, external support, reporting lines, and operating cadence. Together these two documents give leaders a way to think about AI security as a function rather than a single person.

The hiring artifacts translate that architecture into practice. A first-hire decision memo states the organization's current AI security risks, the recommended first archetype, what that hire owns in the first 90 days, what they do not own, and what external support is required during the gap. The memo gives leadership a reasoned decision rather than a title search. A job description template for the chosen archetype follows — mission, responsibilities, required artifacts, interview signals, minimum experience, and explicit non-requirements. Paired with an interview loop map that defines who tests what, which practical exercises apply, and how the scorecard maps to the archetype, this set enables hiring without resorting to keyword pattern-matching.

The operational documents complete the package. A build-vs-buy matrix lists AI security capabilities and marks each as internal, contracted, vendor-supported, or deferred, with a stated reason: frequency, sensitivity, institutional knowledge, specialization, cost, or urgency. This prevents hiring for episodic work while ignoring daily controls. A 30/60/90-day onboarding plan for the first hire includes inventory, stakeholder mapping, top system reviews, first control artifacts, and quick wins. A role without an onboarding plan becomes reactive on day one, which is exactly the wrong posture for a function that is supposed to get ahead of product risk.

Common Failure Modes

One-Person Program Fantasy: Leadership hires one AI security person and assumes the program now exists. The hire becomes a bottleneck for every AI question and cannot produce durable controls. Avoid this by defining the role's first 90 days, explicit non-ownership, and the external support needed for missing archetypes. A person can start a program; they cannot be the whole program indefinitely.

Keyword-Driven Job Description: The JD lists every trending AI security term but does not describe actual work. This attracts candidates who keyword-match and repels practitioners who want a clear mandate. Recover by replacing buzzwords with artifacts and decisions: threat models, tool permission designs, eval gates, model intake, governance evidence, and incident traces.

Wrong First Hire: The company hires a red teamer when the burning need is product review, or hires a governance profile when agents are shipping with broad tool access. This happens when hiring follows market visibility rather than internal risk. Avoid it by mapping current systems and urgent decisions before choosing the archetype.

No Interface With Existing Teams: The AI security hire arrives without clear relationships to AppSec, ML platform, GRC, privacy, procurement, and product engineering. The role then either duplicates work or gets excluded from decisions. Recover by documenting ownership interfaces and release touchpoints during role design, not after onboarding.

Implementation Checklist

• Inventory current and planned AI systems before writing the AI security job description.
• Map required work to the nine AI security archetypes.
• Choose the first hire based on current operational risk, not market hype.
• Write explicit responsibilities, artifacts, and non-responsibilities into the role.
• Define which AI security capabilities will be internal, contracted, vendor-supported, or deferred.
• Build an interview loop that tests the chosen archetype with practical work samples.
• Create a 30/60/90-day onboarding plan tied to real systems and deliverables.
• Define interfaces with AppSec, ProductSec, ML platform, GRC, privacy, procurement, and engineering.

Related Reading

• Handbook chapters: Chapter 1 for discipline scope; Chapter 10 for hiring and assessment design; Chapter 11 for the operating model; Chapter 12 for scorecards and 30/60/90-day templates.
• Field Guide: AI Security Foundations for mental model boundaries; AI-Aware Secure SDLC for product security responsibilities; Red Teaming and Adversarial Evaluations for red-team archetype depth; AI Governance, Risk, and Compliance for evidence leadership.

AI threat modeling almost always starts late. By the time security enters the room, the team has a model provider, a prompt template, a vector index, and a working demo. Decisions about what data the model can see, what tools it can call, and whether retrieved content might carry hostile instructions feel already settled. The question is not whether to do the analysis — it's how to do it effectively even when the design has momentum and the launch date is fixed.

What This Chapter Covers

Threat modeling AI systems means extending familiar security reasoning into systems where model behavior, context, retrieval, tools, and model supply chain all influence risk. This chapter gives practitioners a practical method for threat modeling LLM applications, RAG pipelines, agents, AI-enabled product features, eval workflows, telemetry gaps, and external model dependencies. It solves a real organizational problem: teams that know how to threat model web applications often miss AI-specific trust decisions because they are not visible in ordinary request-response diagrams.

This chapter matters when a team is designing a new AI feature, adding RAG to an existing product, giving an assistant access to tools, changing model providers, launching a copilot, or reviewing an AI feature after an incident. It is especially useful when the room includes mixed stakeholders: AI engineers, software engineers, product managers, security engineers, data owners, GRC, and platform teams. The trigger is simple: if the AI system can see sensitive context, influence a user, retrieve enterprise data, or call a tool, it deserves an AI-aware threat model.

After working through this chapter, you should be able to run a 90-minute AI threat modeling session, enumerate the AI-specific attack surface, identify trust boundaries, rank risks, and produce a control-priority backlog. You should also be able to explain what standard STRIDE still helps with and what it misses. The output is not a whiteboard photo. The output is a populated threat model, a ranked attack-surface list, and a control-priority rubric tied to the system's risk tier.

Core Concepts

STRIDE Still Helps, But It Is Not Enough STRIDE remains useful because AI systems still have spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege risks. The mistake is assuming those categories cover every AI failure clearly. AI systems add non-deterministic outputs, context-based trust decisions, retrieval-time authorization failures, prompt injection, model supply-chain changes, and agent action chains. Use STRIDE as a base layer, then extend it with AI-specific questions.

Context as Attack Surface In AI systems, context is not passive input. It can contain user instructions, system instructions, retrieved documents, conversation history, tool outputs, policies, examples, and hidden application state. Any context segment can influence output, and some segments may carry adversarial instructions or sensitive information. Threat modeling must identify where context comes from, who controls it, how it is labeled, how it is trusted, and what authority it has.

Retrieval Plane as Data Access Layer RAG systems turn retrieval into a security boundary. The threat model must ask whether authorization happens before retrieval, whether chunk metadata preserves permissions, whether tenants share an index, whether deletion propagates to embeddings, and whether source attribution is reliable. If the model receives data the user should not access, output filtering is already too late. Retrieval is not just search; it is a controlled data path.

Agent Action Chains Agent systems change the threat model because model output may become action. A single tool call can write records, send messages, trigger workflows, or modify production systems. A sequence of low-risk calls can combine into a high-risk outcome. Threat modeling agents requires analyzing tool permission class, runtime authorization, approval placement, rollback, auditability, and maximum blast radius.

Evidence-Driven Controls A useful threat model does not stop at risk statements. It identifies controls and the evidence those controls produce. For example, a retrieval authorization control should produce query logs and access decisions. A model intake control should produce provenance and hash records. An agent approval control should produce approver identity and tool-call traces. Controls without evidence are hard to verify and hard to defend during an incident or audit.

The Practitioner's Challenge

The first challenge is that AI threat modeling often starts too late. Product teams may already have a prototype, model provider, prompt template, vector index, and demo workflow before security enters the room. At that point, the hardest design decisions may feel settled. The practitioner must avoid becoming a last-minute blocker while still identifying which assumptions are unsafe enough to require redesign.

The second challenge is mixed vocabulary. AI engineers may speak in terms of embeddings, tools, evals, prompts, and model behavior. AppSec engineers may speak in trust boundaries, authz, injection, secrets, and logging. Product managers may speak in user journeys and launch timelines. A good AI threat modeling session translates across those languages and keeps the group focused on concrete system behavior.

The third challenge is deciding how deep to go. AI systems can be decomposed endlessly: model provider behavior, training data, embeddings, vector stores, tool policies, user roles, streaming, logging, vendor routing, and fallback paths. A session that tries to cover everything equally will fail. The practitioner needs a risk-tiered method that spends time where the system can expose sensitive data, take action, affect customers, or create governance obligations.

How to Approach It

Start with a system walk-through, not a threat list. Ask the product or engineering owner to describe the user journey in plain language. Then draw the technical flow: user input, application server, prompt builder, retrieval, model provider or hosted model, tool layer, output renderer, logs, analytics, and storage. Mark which components are internal, external, user-controlled, generated, retrieved, or privileged.

Next, mark trust boundaries and authority changes. A trust boundary exists when data moves between users, tenants, roles, systems, providers, classification zones, or execution environments. An authority change occurs when text becomes instruction, retrieved data becomes context, model output becomes tool arguments, or generated output becomes a decision. AI threat modeling depends on identifying those authority transitions because many failures occur when low-trust content influences high-trust action.

Then enumerate attack surfaces by layer. For the LLM application layer, ask about prompt assembly, API keys, error handling, streaming, output rendering, caching, and logs. For RAG, ask about ingestion, permissions, metadata, poisoning, tenancy, and citations. For agents, ask about tool scope, approvals, delegation, rollback, and audit logs. For model supply chain, ask about model source, version, format, registry, and promotion. For observability, ask whether incidents can be reconstructed.

Rank risks using impact and control maturity. A prompt injection that changes a harmless summary has different severity from an injection that sends email, leaks tenant data, or modifies production records. A missing log may be medium risk in a toy assistant and critical in an agent that takes irreversible action. Rank based on data sensitivity, action authority, user population, exposure, exploitability, detectability, and reversibility.

End with decisions, not discussion. The session should produce a ranked attack-surface list, control recommendations, release blockers, owners, and evidence requirements. Decide what must be fixed before launch, what can be accepted temporarily, what needs a follow-up design review, and what requires monitoring. A threat model is valuable only if it changes what the team builds, tests, logs, or refuses to ship.

Outputs and Deliverables

The diagrammatic artifacts anchor the threat model. An AI system data-flow diagram covers user inputs, prompt construction, retrieved content, model calls, tool calls, outputs, logs, and vendor routes — each edge labeled with data category, trust level, and whether the content is user-controlled, generated, retrieved, privileged, or externally processed. A trust-boundary and authority map identifies where data crosses tenants, roles, providers, or classification zones, and where authority transitions occur: user text becoming prompt context, retrieved text becoming evidence, model output becoming tool arguments. These authority transitions are where AI-specific risk concentrates and where standard STRIDE exercises are most likely to miss something.

The analytical artifacts give the findings structure and force ranking. A layered attack-surface inventory lists surfaces across the application, retrieval, agent/tool, model supply chain, platform, vendor, and observability layers — each with owner, likelihood, impact, current controls, missing controls, and evidence requirement. A risk-tiered control-priority rubric defines how findings are ranked by data sensitivity, action authority, exposure, reversibility, and evidence quality. A marketing copy generator and an agent that modifies billing records should not share the same gate, and the rubric makes that explicit before the ranking conversation.

The operational artifacts drive action and keep the session from becoming a whiteboard exercise. A release-blocker list names the issues that must prevent launch — missing retrieval authorization, broad agent permissions, no rollback path, no tool-call logging, failed evals, unapproved model changes — and identifies who can accept them as explicit risk decisions. A control evidence plan specifies what artifact proves each major control operated, converting the threat model into a future audit and incident response asset. A 90-minute facilitation agenda lets practitioners run the session consistently with mixed audiences. These documents together convert a threat model session into work on the backlog rather than a photo of a whiteboard that no one updates.

Common Failure Modes

Whiteboard Without Backlog: The team has a lively session but produces no tickets, owners, or release decisions. This happens when facilitation emphasizes brainstorming over output. Avoid it by reserving time at the end for ranked controls, blockers, and owners. A threat model that does not alter the backlog is a conversation, not a control.

Prompt-Only Threat Modeling: The session focuses on jailbreaks and ignores retrieval, tools, model artifacts, logs, and release gates. This happens because prompt attacks are easy to demo and understand. Recover by using the layered attack-surface inventory and forcing the group to review each layer. Prompt security is one section of the model.

Generic STRIDE Reuse: The team runs a standard STRIDE exercise without adapting questions for context, model behavior, retrieval, or agents. This produces familiar findings while missing AI-specific failures. Avoid it by adding authority transitions, retrieval authorization, tool action, model update, and eval evidence to the template. Keep STRIDE, but extend it.

No Risk Tiering: Every issue receives similar treatment, so the team either overreacts or ignores the whole output. AI systems vary widely in severity. A marketing copy generator and an agent that changes customer billing records should not share the same gate. Use data sensitivity and action authority to scale controls.

Implementation Checklist

• Draw the AI system flow from user input to model call to output and downstream effects.
• Identify every trust boundary and authority transition in the system.
• Enumerate attack surfaces across application, retrieval, agent, model supply chain, platform, vendor, and observability layers.
• Identify which AI-specific controls must block release if absent or failed.
• Rank risks by data sensitivity, action authority, exposure, reversibility, and evidence quality.
• Assign each control recommendation to an owner and backlog item.
• Define what evidence proves each major control operated.
• Convert at least one threat model finding into an eval, test, log, or release gate.

Related Reading

• Handbook chapters: Chapter 4 for prompt injection and RAG security; Chapter 5 for agent and tool-calling security; Chapter 6 for model supply chain; Chapter 7 for evals and red-team evidence; Chapter 11 for operating model integration.
• Field Guide: Prompt Injection and Context Security for context threats; RAG Security for retrieval-plane analysis; Agent Security for delegated action; Secure AI Architecture Design for design-level trust placement.

The two failure modes that matter most in production RAG systems are not exotic. The first is prompt injection through retrieved content: a document the model was supposed to read becomes an instruction the model follows. The second is retrieval authorization failure: the model receives data the user was never allowed to see, and output filtering is already too late. Neither is an edge case; both are the default outcome of a RAG system where retrieval was designed for relevance and authorization was added later.

What This Chapter Covers

This chapter covers the practical controls required to secure prompt-driven and retrieval-augmented AI systems. It explains direct prompt injection, indirect prompt injection, context poisoning, retrieval-time authorization, vector-store tenancy, chunk metadata, citation integrity, deletion propagation, and validation testing. The organizational problem it solves is a common one: a product team builds a useful RAG assistant, security arrives late, and everyone discovers that the system retrieves the right documents for relevance but not necessarily the right documents for authorization.

This chapter is relevant when an organization is building an internal knowledge assistant, customer-support copilot, developer documentation chatbot, analyst assistant, legal or compliance search assistant, or any AI feature that combines user input with retrieved context. It is also relevant when a team already has a RAG prototype and now needs to answer customer or auditor questions about data boundaries, source attribution, prompt injection, and deletion behavior. The chapter is written for AppSec, ProductSec, AI engineering, platform engineering, and security architecture teams who need a shared control model.

After working through this chapter, you should be able to review a RAG design, identify whether retrieval is authorized before generation, classify direct and indirect prompt injection paths, design chunk-level metadata controls, write practical RAG security tests, and explain why retrieval is a data access decision rather than a search decision. You should also be able to separate prompt injection defenses from retrieval authorization controls instead of treating them as one generic "LLM safety" concern.

Core Concepts

Direct and Indirect Prompt Injection Direct prompt injection occurs when a user intentionally gives the model instructions that conflict with the application's intended behavior. Indirect prompt injection occurs when hostile instructions arrive through content the application retrieves or processes, such as documents, web pages, emails, tickets, calendar entries, or tool outputs. Direct injection is easier to see because it appears in the user turn. Indirect injection is more dangerous in many production systems because the application often treats retrieved content as evidence, not as a possible attacker-controlled instruction channel. The defense must assume that some context will be adversarial, even when it comes from an internal source.

Context Poisoning Context poisoning happens when untrusted content changes the model's behavior over a session, workflow, or multi-step process. The poisoning may be explicit, such as "ignore prior instructions," or subtle, such as false policy claims, fake source authority, or staged assumptions that alter later outputs. In RAG systems, poisoned content may live in the knowledge base and activate only when retrieved for a specific query. In agentic systems, poisoned context can influence tool arguments or approval narratives. The control objective is not to make the model perfectly immune; it is to reduce the authority of untrusted context and validate the actions or outputs that follow.

Retrieval-Time Authorization Retrieval-time authorization is the principle that a user's permissions must be checked before content enters the prompt. Post-generation filtering cannot compensate for a bad retrieval decision because the model has already processed the unauthorized content. It may summarize it, paraphrase it, infer from it, or leak it through partial output even if a final filter blocks exact strings. Retrieval should apply tenant, role, document, classification, purpose, and freshness constraints before ranking or context assembly. If the user cannot access the source record, the model should not receive the chunk.

Vector-Store Tenancy and Metadata Integrity Vector stores do not enforce business boundaries by default. A shared index can be acceptable if metadata filters are mandatory, correct, and tamper-resistant, but it creates different failure modes from tenant-namespaced or physically separated indexes. Chunk metadata should preserve source ID, tenant, owner, classification, ACL, ingestion timestamp, deletion status, and version. If metadata is stripped during chunking or treated as an optional query hint, authorization becomes fragile. Secure RAG depends on metadata integrity as much as embedding quality.

Source Trust Tiers and Citation Integrity Not every source should influence the model with the same authority. System instructions, developer instructions, user questions, internal policy documents, wiki pages, customer uploads, tool outputs, and web content all need different trust semantics. Internal knowledge-base content may be data-safe for summarization without being instruction-safe for controlling the assistant. Citation integrity means the answer's claims can be traced to retrieved chunks that actually support the response. It is both a user trust mechanism and an incident response artifact.

The Practitioner's Challenge

The political challenge is that RAG systems often prove value before they prove security. Relevance demos are compelling: the assistant finds documents, answers questions, and reduces search friction. Authorization, metadata integrity, deletion propagation, and injection testing feel like launch blockers after the product already works. The practitioner has to reframe the conversation: the system does not truly work if it finds the right answer for the wrong user.

The structural challenge is ownership across teams. Search or AI engineering may own embeddings and retrieval quality. Product engineering may own the application and prompt builder. Identity teams may own permissions. Data owners may own source systems. Security may own threat modeling and validation. A RAG security failure often emerges between these teams, especially when permissions in the source system do not map cleanly to chunks in the vector index.

The technical challenge is that relevance and authorization pull in different directions. Retrieval wants broad semantic recall; security wants strict filtering and traceable source boundaries. Chunking can improve model performance while weakening permission fidelity. Summarization can improve usability while weakening citation integrity. The practitioner must design controls that preserve enough retrieval quality without treating the vector database as a permissionless semantic soup.

How to Approach It

Start with the source systems. Identify every corpus that can feed the RAG system: documents, tickets, wikis, email, customer records, code repositories, policies, uploaded files, or vendor content. For each source, record the owner, data classification, tenant model, permission model, deletion behavior, and ingestion path. Do not start with the vector database; start with the data authority that the vector database must preserve.

Next, map the ingestion pipeline. Track how documents become chunks, how chunks become embeddings, which metadata is attached, where the records are stored, and how updates or deletions propagate. Verify that source IDs and authorization metadata survive chunking. If a chunk cannot be traced back to an authoritative source and permission state, it should not be eligible for production retrieval.

Then design retrieval as an authorization workflow. The query should carry user identity, tenant, role, purpose, and request context into the retrieval layer. Mandatory filters should reduce the candidate set before similarity ranking. Metadata policy should be enforced by code or platform constraints, not by convention. If a required filter is missing or ambiguous, the retrieval layer should fail closed.

Separate source trust from source relevance. A highly relevant document may still be low-trust, user-generated, stale, or instruction-unsafe. Treat retrieved content as evidence for the answer, not as policy for the system. Context formatting should label source, classification, and role clearly, but formatting is not enough. Output validation, citation checks, and tool-policy controls must enforce the boundaries that the model cannot reliably maintain by itself.

Build the validation plan in three lanes. The first lane tests direct prompt injection through user turns. The second tests indirect injection through retrieved content, tool outputs, and imported documents. The third tests retrieval authorization independently of prompt injection by verifying that unauthorized chunks cannot enter context at all. These lanes should run separately because a system can pass one and fail another.

End with operational evidence. RAG security should produce ingestion records, metadata schemas, authorization test results, retrieval logs, citation validation reports, deletion propagation tests, and injection regression cases. Store those artifacts where product, security, GRC, and incident response can use them. A RAG control that cannot be evidenced will be hard to defend when a customer asks how the assistant avoids cross-tenant leakage.

Outputs and Deliverables

The core design artifacts are the RAG data-flow map, source inventory, and chunk metadata schema. The data-flow map shows how source records move through ingestion, chunking, embedding, indexing, retrieval, prompt assembly, generation, citation, logging, and deletion. The source inventory names each corpus, owner, classification, permission model, update cadence, and deletion behavior. The chunk metadata schema defines the fields required for secure retrieval, such as source ID, tenant, ACL reference, classification, ingestion time, version, deletion marker, and trust tier.

The enforcement artifacts are the retrieval authorization policy, vector-store tenancy decision, and RAG security checklist. The authorization policy explains which filters must be applied before similarity ranking and what happens when user identity, tenant, classification, or ACL state is missing. The tenancy decision records whether the system uses shared indexes, tenant namespaces, separate indexes, or separate stores, and why that choice is acceptable for the data involved. The checklist gives reviewers a concrete way to test ingestion, permissions, metadata, citations, deletion, logging, and prompt injection.

The validation and evidence artifacts are the prompt injection test set, retrieval authorization test set, citation integrity report, and deletion propagation test record. The prompt injection tests should include direct user-turn attacks and indirect attacks embedded in documents, tickets, emails, and web content. The retrieval authorization tests should prove unauthorized chunks do not enter context, independent of whether the model would reveal them. Citation and deletion tests show whether answers can be traced to valid sources and whether removed data stops appearing in retrieval.

Common Failure Modes

Relevance-First Retrieval: The system ranks across the broadest possible corpus and adds authorization later. It looks good in demos because it finds semantically strong answers. It fails security review because high-privilege context can reach low-privilege sessions. Recover by enforcing mandatory authorization filters before ranking.

Internal Source Overtrust: The team assumes internal documents cannot contain hostile instructions. This fails when wikis, tickets, shared drives, support cases, and imported vendor text contain user-generated or low-review content. Treat internal sources as data-safe only for their intended purpose, not instruction-safe. Use trust tiers and indirect injection tests.

Metadata Loss During Chunking: Permissions and classification labels exist at the source document level but disappear when the document becomes chunks. The vector store then cannot enforce policy accurately. Recover by preserving source IDs and ACL references on every chunk and by testing permission changes after ingestion.

Citation Theater: The system displays citations that look authoritative but are not tied tightly to retrieved evidence. This happens when the model generates citations or when attribution is assembled after the answer. Recover by binding citations to retrieved chunk IDs and validating that claims are supported by the cited source.

Implementation Checklist

• Inventory every source corpus that feeds the RAG system and record owner, classification, permission model, and deletion behavior.
• Define the chunk metadata schema required for authorization, traceability, deletion, and citation integrity.
• Enforce tenant, role, document, classification, and purpose filters before similarity ranking.
• Decide and document the vector-store tenancy model for each data classification.
• Label context by source trust tier and prevent retrieved content from acting as system instruction.
• Build separate tests for direct prompt injection, indirect prompt injection, and retrieval authorization.
• Verify deletion propagation from source record to chunk, embedding, cache, and citation.
• Store retrieval logs and citation evidence so incidents can be reconstructed.

Related Reading

• Handbook chapters: Chapter 3, Threat Modeling AI Systems; Chapter 5, Agent and Tool-Calling Security; Chapter 7, Evals, Red Teaming, and Evidence; Chapter 8, Governance-to-Engineering Evidence.
• Field Guide: Prompt Injection and Context Security; RAG Security; LLM Application Security; Privacy and Data Protection; Incident Response and AI Observability.

The security model for agents breaks down quickly when you follow one question to its conclusion: what is the maximum blast radius of one confused or compromised model call? For a text assistant, the answer may be a bad output. For an agent with write access to email, source code, cloud resources, issue trackers, calendars, and customer records, the answer can be an organization-wide incident triggered by a single injected instruction in a retrieved document. The gap between those two answers is the entire scope of agent security.

What This Chapter Covers

This chapter covers practical security engineering for AI systems where model output becomes tool calls, tool calls become state changes, and state changes affect real users, data, infrastructure, or business processes. It explains delegated action, tool permission design, runtime authorization, approval gates, action chaining, delegation chains, sandboxing, rollback, reversibility, audit trails, and blast-radius limits. The organizational problem it solves is that agent prototypes often grant tools to models before anyone defines what the model is allowed to do, what requires approval, or what evidence will exist when something goes wrong.

This chapter is relevant when a team gives an LLM access to internal APIs, SaaS connectors, email, code repositories, cloud consoles, ticketing systems, browsers, file systems, command execution, calendars, databases, or workflow automation. It is especially relevant when the product language shifts from "assistant" to "agent," "autopilot," "copilot," "workflow automation," or "AI employee." The reader may be an AppSec engineer reviewing a tool-calling feature, a platform engineer designing an agent runtime, a red teamer testing delegated action, or a security architect setting policy for agentic systems.

After working through this chapter, you should be able to classify tool permissions, design runtime authorization around actual capabilities, decide where human approval matters, evaluate action chains, reason about multi-agent delegation, define rollback requirements, and specify audit logs for forensic reconstruction. You should also be able to challenge weak arguments such as "the tool description says read-only" or "the human approved it" when those claims are not backed by enforceable policy and useful context.

Core Concepts

Delegated Action Model Agent security starts with the delegated action chain: user request becomes model reasoning, model reasoning becomes tool arguments, tool execution changes state, and the result may influence another model call. Each transition changes the risk. A generated answer can be wrong without changing the world; a tool call can send email, modify records, create cloud resources, or delete data. The security review should trace the full path from prompt to side effect, not just inspect the model response.

Tool Permission Design Tool permissions should be scoped by resource target, action type, tenant boundary, user role, time window, quota, and reversibility. A tool called "send_message" is not one permission; sending a draft to the current user, sending an email to a customer, posting in a public channel, and notifying every administrator are different risk classes. Least privilege means the credential and policy wrapper enforce the narrowest action needed for the workflow. Good tool design makes dangerous action impossible by default rather than relying on the model to avoid it.

Runtime Authorization Tool labels and descriptions are not enforcement. If a tool is described as read-only but the underlying credential can write, the system is write-capable. Runtime authorization checks the acting user, agent identity, tenant, resource, action, arguments, current context, and policy before execution. The policy should live outside the model so an injected instruction cannot redefine what is allowed. The model can propose an action; the runtime decides whether the action is permitted.

Approval Gate Design Human approval is valuable when it is rare enough to receive attention, informative enough to support judgment, and placed before actions that are irreversible, externally visible, high-volume, destructive, or privileged. Approval becomes ceremony when every trivial action prompts a click, when the approver lacks context, or when the prompt hides the true target and arguments. A useful approval request shows what will happen, why the agent proposes it, which evidence supports it, what resources are affected, whether it can be undone, and what policy triggered approval. Approval is not a magic shield; it is a control that needs design.

Blast Radius as Architecture Constraint Blast radius is the maximum damage a compromised, confused, or misled agent can cause before another control stops it. It must be designed before implementation because after an incident the system has already exercised its available authority. The blast radius of a tool depends on credentials, resource scope, action scope, quotas, environment access, network access, and action chaining. Prompt patches do not reduce the authority already granted to a tool. Architecture does.

The Practitioner's Challenge

The political challenge is that agents are often sold internally as productivity accelerators. Teams want tools connected quickly because the demo value is immediate: the agent files tickets, updates documents, searches systems, drafts messages, and completes workflows. Security friction can sound like resistance to automation. The practitioner has to reframe controls as what makes automation deployable, not what makes it slower.

The structural challenge is ownership. The model team may own orchestration, platform engineering may own the runtime, product engineering may own user experience, IT may own SaaS connectors, security may own policy, and business teams may own the workflows. An unsafe tool chain can emerge because every team owns a piece and no one owns the end-to-end authority model. Agent security requires a single view of what the agent can do across systems.

The technical challenge is composition. A single read operation may be low risk, but a sequence of reads can collect enough context for disclosure. A draft action may be low risk until paired with a send action. A code generation tool may be manageable until paired with repository write access and CI triggers. The practitioner must analyze action chains rather than individual tool calls in isolation.

How to Approach It

Start with a tool inventory. List every tool, connector, API, execution environment, and sub-agent the system can use. For each one, record the underlying credential, action class, resource scope, tenant scope, reversibility, external visibility, data classification, rate limit, and owner. Do not accept the tool's friendly name or manifest description as the security description. Inspect what the credential can actually do.

Next, classify action risk. Separate read-only, write, destructive, irreversible, external communication, privilege-changing, financial, production-modifying, and code-executing actions. Assign different baseline requirements to each class. Read-only actions may require logging and scope limits. External messages may require approval. Destructive actions may require stricter authorization, delay, dual approval, or prohibition. Code execution may require sandboxing and egress controls.

Then design runtime authorization around the user and workflow. Decide whether the agent acts as the user, as itself, or as a service account with delegated authority. For each tool call, enforce policy using user identity, tenant, resource target, action type, arguments, and workflow state. Avoid broad static credentials when possible. If the agent acts through a service account, the policy wrapper must reintroduce user-level and tenant-level constraints.

Design approval gates only where they change outcomes. Identify irreversible or externally visible actions, broad writes, destructive changes, privilege changes, financial transactions, production changes, and sensitive disclosures. For those actions, build approval screens that show the proposed operation, target resources, source evidence, risk reason, reversibility, and alternatives. If approvers cannot understand what they are approving, the gate is theater.

Analyze action chains and delegation paths. Walk through multi-step workflows and ask what a malicious document, tool output, or user prompt could steer the agent to do. Identify combinations that create higher risk than any individual tool. If one agent can call another, define whether authority transfers, whether the child agent inherits context, what logs link the chain, and which policy engine makes decisions.

End by designing auditability and rollback. Define required log fields before launch: user, tenant, agent identity, model version, prompt/context references, tool name, arguments, authorization decision, approval decision, result, side effect, reversibility flag, and parent trace ID. For each action class, decide whether rollback is possible and how it is executed. If an action is irreversible, require stronger prevention before it runs.

Outputs and Deliverables

The core design deliverables are the agent tool inventory, tool permission matrix, and blast-radius worksheet. The inventory names every connector, API, code runner, browser action, sub-agent, and workflow integration available to the agent. The permission matrix classifies each tool by action type, credential, resource scope, tenant boundary, data classification, rate limit, and owner. The blast-radius worksheet translates those details into a practical question: if this tool is misused once, what is the worst plausible outcome?

The enforcement deliverables are the runtime authorization policy, approval gate design, and sandboxing profile. The runtime policy defines which identity the agent acts under, which checks occur before execution, what arguments are allowed, and what conditions fail closed. The approval design specifies which actions require approval, what context the approver sees, and what evidence the decision creates. The sandboxing profile defines filesystem access, network egress, credential exposure, execution limits, package installation rules, and isolation boundaries for code-executing or browser-driving agents.

The operational deliverables are the agent audit schema, rollback plan, and agent abuse test plan. The audit schema ensures every action chain can be reconstructed from user request to model call to tool execution to side effect. The rollback plan distinguishes reversible actions, compensating actions, and irreversible actions that require prevention rather than recovery. The abuse test plan covers prompt injection through retrieved content, unexpected tool arguments, confused-deputy paths, approval bypass, chained low-risk actions, and delegation drift.

Common Failure Modes

Manifest Trust: The team trusts tool names, descriptions, or manifest labels as if they enforce permissions. This happens when engineering treats the LLM tool interface as the security boundary. Recover by inspecting the underlying credential and placing runtime policy outside the model. A read-only description attached to a write-capable token is not read-only.

Approval Fatigue: The system asks humans to approve too many low-context actions. Approvers learn to click through because the requests are frequent and uninformative. Avoid this by reserving approval for meaningful risk thresholds and showing enough context to make a real decision. A good approval gate should be rare, specific, and evidence-rich.

Action Chain Blindness: The team reviews tools individually and misses the risk created by combining them. Reading a record, summarizing it, drafting a message, and sending it may become a disclosure path. Recover by threat modeling workflows end to end and testing sequences, not just single calls. Tool composition is where agent risk often becomes serious.

Rollback Assumption: The team assumes harmful actions can be undone later. Some actions cannot be fully reversed: external emails, data disclosures, financial transactions, privilege changes, and customer-visible updates may leave permanent effects. Recover by classifying reversibility before launch and applying stronger approval or prohibition to irreversible actions. Rollback is not a substitute for prevention.

Implementation Checklist

• Inventory every tool, connector, API, code runner, browser action, and sub-agent available to the agent.
• Classify each tool by read, write, destructive, irreversible, external, privilege-changing, code-executing, or production-modifying action.
• Verify the underlying credential and API permissions instead of trusting tool labels or descriptions.
• Define runtime authorization checks for user, tenant, resource, action, arguments, and workflow state.
• Design approval gates for irreversible, external, destructive, broad-scope, or privileged actions.
• Analyze action chains for compound risk across multiple low-risk tools.
• Define sandbox limits for code execution, filesystem access, network egress, and credential exposure.
• Implement audit logs that reconstruct user request, model call, tool arguments, policy decision, approval, result, and side effect.

Related Reading

• Handbook chapters: Chapter 3, Threat Modeling AI Systems; Chapter 4, Prompt Injection and RAG Security; Chapter 7, Evals, Red Teaming, and Evidence; Chapter 11, Building the Operating Model.
• Field Guide: Agent Security; Prompt Injection and Context Security; Secure AI Architecture Design; Incident Response and AI Observability; LLM Application Security.

Organizations that would never deploy a dependency without reviewing its source, checking its hash, and verifying its license regularly deploy model weights downloaded from public hubs with none of those checks. The oversight is not usually negligence. It is category error. The team that owns model deployment thinks in terms of performance and inference cost, not supply-chain trust, and model supply chain security exists to close that gap.

What This Chapter Covers

This chapter covers the controls needed to manage model artifacts from discovery through production deployment. It explains model provenance, artifact integrity, unsafe serialization formats, public hub risk, base model lineage, license compliance, registry controls, intake review, approval workflows, version pinning, and CI/CD integration. The organizational problem it solves is that models are often treated like data files or performance assets when they should be treated like production supply-chain components with security, legal, operational, and governance implications.

This chapter is relevant when a team downloads models from Hugging Face, Civitai, GitHub, vendor portals, internal research teams, partner deliveries, or model marketplaces. It also applies when the organization fine-tunes base models, packages adapters, promotes models through MLflow or a cloud registry, deploys local models through inference servers, or uses model artifacts inside applications. The reader may be an ML platform engineer, product security engineer, AppSec practitioner, AI security engineer, security architect, or governance lead who needs model deployment to become reviewable and repeatable.

After working through this chapter, you should be able to define a model intake process, verify artifact integrity, distinguish unsafe loading risk from broader provenance risk, design registry promotion controls, evaluate public hub trust, and explain why fine-tunes inherit risk from base models. You should also be able to write a model change management policy that operations teams can follow without turning every model update into a bureaucratic emergency.

Core Concepts

Model Provenance Model provenance answers where the model came from, who created it, what it was trained or fine-tuned from, what data influenced it, what license applies, and who approved it for use. Provenance is not only a model card link. A useful provenance record identifies publisher, source URL, exact version or commit, artifact hash, base model, adapter lineage, training or fine-tuning method where known, intended use, limitations, and owner. Without provenance, teams cannot investigate behavior, defend customer claims, or prove that the artifact in production matches the artifact reviewed. Provenance must be recorded before production promotion, not reconstructed during an incident.

Artifact Integrity Artifact integrity proves that the model artifact loaded in production is the artifact that was reviewed and approved. The core controls are cryptographic hashes, signatures where available, immutable storage, registry promotion workflows, and deployment pinning. A mutable branch, tag, or "latest" reference is not a stable production dependency. Integrity verification should occur before model loading and again at promotion boundaries. The goal is to prevent silent drift, substitution, and accidental deployment of unreviewed artifacts.

Unsafe Serialization Formats Some model and ML artifact formats can execute code during loading. Pickle-based artifacts are the classic example, but the broader issue includes Python object serialization, custom loaders, model packages that execute repository code, and preprocessing artifacts that run as part of inference. Safer formats such as safetensors reduce code execution risk for weights, but format safety is only one control. A safetensors file can still have unknown provenance, an incompatible license, poor eval evidence, or inherited behavioral risk from a base model.

Model Registries as Control Points A model registry becomes a governance control only when it enforces metadata, access, approval, versioning, and promotion rules. If the registry is just a folder with a UI, it stores artifacts but does not control them. A production-ready registry entry should include owner, version, source, hash, base model lineage, license, allowed use, eval evidence, approval status, deployment targets, and rollback version. Promotion from experimental to staging to production should require checks that are visible and auditable. The registry is where model supply-chain evidence becomes operational.

Base Model Lineage A fine-tuned model inherits properties from its base model: license obligations, known limitations, safety characteristics, possible memorization, benchmark weaknesses, and upstream vulnerabilities. Approving a fine-tune without approving the base model is incomplete. Adapter-based systems make this more complex because the deployed behavior may depend on base model, adapter, tokenizer, prompt template, and serving configuration together. Model lineage should record the full chain needed to reproduce and assess the deployed artifact.

The Practitioner's Challenge

The political challenge is velocity. AI teams experiment quickly, and model selection often changes during product iteration. Security review can be perceived as slowing down research or blocking performance improvements. The practitioner has to separate experimentation from production promotion. Exploration can remain flexible, but production deployment needs provenance, integrity, license review, eval evidence, and rollback planning.

The structural challenge is fragmented ownership. Research may choose the model, ML platform may host it, product engineering may integrate it, legal may care about license, GRC may need evidence, and security may own supply-chain review. If no one owns the model intake path end to end, artifacts move from notebooks to production through informal trust. Model supply-chain security requires an explicit handoff from experimentation to controlled deployment.

The technical challenge is that model artifacts are not always self-describing. A checkpoint may not reveal its training data, publisher confidence, base lineage, or license implications. Some artifacts require custom code to load, and some repositories mix model weights with scripts, tokenizers, configs, adapters, and examples. The practitioner must design a process that handles incomplete information without pretending uncertainty is the same as approval.

How to Approach It

Start by separating model discovery from production intake. Teams should be able to experiment, but production candidates must enter a formal intake path. Define the trigger: any model, adapter, embedding model, reranker, tokenizer, or preprocessing artifact that will influence production behavior must receive an intake record. The intake record should name the owner, intended use, source, version, artifact hash, base lineage, license, and deployment target.

Next, define approved artifact sources. Public hubs may be allowed for discovery but not direct production pulls. A safer pattern is to review the artifact, record metadata, verify hashes, mirror it into controlled storage, and deploy from the internal registry or artifact repository. For vendor-provided models, require delivery metadata, checksum or signature, license terms, security notes, and model change notice expectations. The production system should not depend on a mutable remote artifact.

Then establish format and loading rules. Decide which formats are allowed, which require sandboxing, and which are prohibited. For example, safetensors may be allowed for weights, while pickle or custom Python loaders require isolation or are blocked for production. If repository code must be executed to load a model, treat that code as a dependency requiring review. Document the loader path, not just the artifact name.

Build registry promotion as the control point. Experimental artifacts can exist, but production promotion should require required metadata, integrity verification, license review, eval evidence, security review for high-risk deployments, and rollback target. Access controls should prevent arbitrary users from promoting models to production. Registry events should feed audit logs and release evidence.

Integrate checks into CI/CD and deployment. Promotion or deployment should verify hashes, reject mutable references, check required metadata, enforce approved formats, confirm eval evidence, and ensure the deployment references an approved registry version. These checks reduce reliance on manual memory. They also make model changes visible to release processes that already govern application code.

End by designing change management that teams can actually follow. Not every model update needs the same depth of review. Risk-tier updates by data sensitivity, action authority, user population, deployment exposure, and reversibility. A low-risk internal summarizer may need lightweight checks, while a customer-facing agent or regulated decision-support model needs stronger approvals, evals, and notice. The process should scale with risk.

Outputs and Deliverables

The core governance artifacts are the model intake record, model provenance record, and base lineage map. The intake record captures why the model is being considered, where it came from, who owns it, and what deployment it will influence. The provenance record ties source, publisher, exact version, hash, license, base model, adapter chain, and approval status into one reviewable artifact. The base lineage map makes inherited risk visible, especially when fine-tunes, adapters, tokenizers, and serving configurations combine to create production behavior.

The operational control artifacts are the model registry promotion policy, allowed format policy, and artifact integrity verification workflow. The promotion policy defines required metadata, approval stages, access control, rollback expectations, and evidence gates for moving models into production. The format policy distinguishes safe, restricted, sandbox-only, and prohibited loading paths. The integrity workflow defines when hashes or signatures are checked, where approved artifacts are stored, and how deployments prove they loaded the approved version.

The release and assurance artifacts are the model change management policy, license review record, model deployment manifest, and supply-chain CI/CD checks. The change policy tells teams what must re-run when a base model, fine-tune, embedding model, tokenizer, or serving configuration changes. The license record documents commercial rights, attribution requirements, restrictions, and output implications. The deployment manifest records the exact model version, artifact hash, registry ID, eval evidence, owner, and rollback version used by a production service.

Runtime, Host, and Cluster Boundary

Model supply chain controls do not end when an artifact enters the registry. The artifact still has to load and run somewhere, and the runtime environment can quietly become the real security boundary. A model-serving host may hold model weights, provider keys, vector-store credentials, prompt logs, cached outputs, customer context, and telemetry. A training or inference cluster may mix workloads with different trust levels. A notebook may combine code execution, data access, package installation, and production-adjacent credentials. These are not separate from AI security; they are where the approved artifact becomes a live system.

For production and production-adjacent AI workloads, the operating model should require a model-serving environment review before launch. That review names the host or managed service, container image, model artifact, data categories, runtime credentials, network egress, logging policy, patch cadence, workload identity, and emergency disablement path. If the system uses GPUs, the review should also state the isolation model: dedicated node, namespace, tenant pool, shared device, managed service boundary, or other arrangement. The question is not whether the GPU is special. The question is whether workloads with different trust levels can observe, affect, starve, or escape each other.

Secrets deserve separate treatment. Provider keys, registry tokens, vector-store credentials, tool credentials, and telemetry keys should not be baked into images, notebooks, prompt templates, cached outputs, or client-visible configuration. Prefer runtime identity, short-lived credentials, secret managers, and scoped service accounts. If the model-serving process can call tools or retrieve customer context, its credential scope should match the workflow rather than the convenience of the platform.

Trusted execution environments and confidential computing may support specific threat models, but they should not be presented as general proof that an AI system is secure. Use them when the risk model involves provider visibility, memory exposure, attestation, or protected key release, and record what boundary they actually protect. They do not replace retrieval authorization, model intake, unsafe-loader policy, endpoint rate limits, logging, or incident response.

The evidence artifacts for this layer are practical: Hardware Isolation Review, GPU and Host Isolation Checklist, Model Serving Environment Review, Cluster Access Review, Inference Secrets Review, patch records, workload identity maps, and emergency rollback or disablement logs. A team should be able to prove which artifact ran, where it ran, which credentials were available, who could access the environment, and how the service would be contained during an incident.

Common Failure Modes

Public Hub to Production: A service pulls directly from a public hub at deployment or startup. This happens because it is convenient and common in examples. It fails because the organization cannot guarantee artifact stability, provenance, or review status. Recover by mirroring approved artifacts into controlled storage and deploying only pinned internal versions.

Format Safety Confusion: A team treats safetensors or another safer format as complete supply-chain security. Format safety reduces one class of code execution risk, but it does not establish provenance, license compliance, eval evidence, or approval. Recover by treating format as one field in the intake record, not the whole review. The model still needs lineage and promotion controls.

Registry-as-Storage: The organization has a model registry but no required metadata, approvals, access controls, or promotion workflow. Artifacts look official because they are in the registry, but anyone can upload or promote them. Recover by turning the registry into a gate: required fields, restricted promotion, immutable versions, evidence links, and audit logs.

Invisible Base Model Risk: A fine-tune is approved based on its immediate performance while the base model is unknown or unapproved. This happens when teams review the final artifact but not the lineage. Recover by requiring base model documentation and license review before approving derived artifacts. A fine-tune cannot be more trustworthy than its unresolved base chain.

Runtime Boundary Blind Spot: The model is approved, but the serving host exposes broad credentials, weak egress controls, shared GPU access, stale images, or unreviewed notebook paths. Recover by reviewing the serving environment as part of production promotion and requiring host, secret, workload, and patch evidence before launch.

Implementation Checklist

• Define the trigger that requires a model, adapter, tokenizer, embedding model, or preprocessing artifact to enter production intake.
• Require source, version, publisher, owner, intended use, license, base lineage, and artifact hash in every model intake record.
• Prohibit mutable production references such as latest tags, unpinned branches, or uncontrolled hub downloads.
• Define allowed, restricted, sandbox-only, and prohibited artifact formats.
• Mirror approved public or vendor artifacts into controlled internal storage before production deployment.
• Configure the model registry to enforce metadata, promotion approvals, access control, immutable versions, and rollback references.
• Integrate hash verification, metadata checks, format checks, and eval evidence checks into promotion or deployment.
• Create a risk-tiered change management policy for model, adapter, embedding, tokenizer, and serving configuration updates.
• Review model-serving hosts, cluster access, runtime credentials, GPU isolation, egress, patch cadence, and emergency disablement before production use.

Related Reading

• Handbook chapters: Chapter 3, Threat Modeling AI Systems; Chapter 7, Evals, Red Teaming, and Evidence; Chapter 8, Governance-to-Engineering Evidence; Chapter 11, Building the Operating Model.
• Field Guide: Model Supply Chain Security; MLOps Platform Security; AI-Aware Secure SDLC; Vendor Risk and AI Procurement; Secure AI Architecture Design.

Most AI red team exercises produce a report. The report describes what the team found, maybe includes some screenshots, and recommends fixes. Then the assessed team decides which findings matter. That is not adversarial evaluation; it is advisory with a dramatic aesthetic. The difference between a red team exercise and an adversarial control is whether the findings produce regression tests, whether those tests block future releases, and whether closure requires evidence rather than conversation.

What This Chapter Covers

This chapter covers how to turn AI evals and red teaming into repeatable security controls. It explains the difference between automated evaluations and human red-team exercises, how severity rubrics should be defined before testing begins, how prompt attack libraries become maintained assets, how red-team findings become regression tests, and how eval outputs become release and audit evidence. The organizational problem it solves is that many AI security tests are treated as one-time events instead of operating controls.

This chapter is relevant when a product team is preparing to launch an AI feature, when a model update changes system behavior, when governance asks for evidence, when a customer asks whether prompt injection or unsafe output has been tested, or when a red team has delivered findings that now need closure. It is also relevant when teams are building CI/CD gates for model, prompt, retrieval, or tool changes. The chapter is written for AI security engineers, red teamers, product security teams, AI platform owners, and GRC leads who need adversarial testing to produce durable evidence.

After working through this chapter, you should be able to design an eval suite tied to production behavior, scope a human red-team exercise, write severity definitions, convert findings into regression tests, define closure criteria, and preserve evidence in a form useful for release gates, audits, and customer assurance. You should also be able to identify benchmark gaming and distinguish real security testing from impressive but non-operational demos.

Core Concepts

Evals as Release Controls An eval becomes a control when it has an owner, expected behavior, severity, pass/fail threshold, execution cadence, and release consequence. A test that runs after launch and produces a dashboard is useful, but it is not a release gate unless failure changes the shipping decision. AI evals should cover the deployed system surface, not just raw model behavior. For a RAG assistant, that means testing retrieval, context assembly, citations, and output behavior together. For an agent, it means testing tool arguments, authorization decisions, approvals, and side effects.

Human Red Teaming Human red teams are strongest where judgment, creativity, and chained reasoning matter. They discover failure modes that automated suites do not yet represent: indirect injection through realistic documents, policy bypass through workflow context, multi-step agent abuse, or unsafe behavior emerging from user interaction. Human red teaming should be scoped, severity-rated, and evidence-rich. Its most valuable output is not only the report; it is the new set of test cases, controls, and architectural questions the exercise creates.

Severity Rubrics Before Testing Severity definitions must exist before findings are delivered. Critical, high, medium, low, informational, and out-of-scope categories should be tied to impact, exploitability, affected users, data sensitivity, action authority, reversibility, and control failure. If severity is negotiated after the finding appears, the assessed team can unconsciously downgrade uncomfortable results. A pre-agreed rubric makes closure disciplined and reduces political friction. It also lets leadership understand which failures block release.

Prompt Attack Libraries A prompt attack library is a maintained body of adversarial scenarios, payloads, expected behaviors, and reproduction notes. It should cover direct prompt injection, indirect prompt injection, context poisoning, jailbreak chains, retrieval poisoning, policy bypass, unsafe output, sensitive disclosure, and tool misuse. The library should be versioned and mapped to product surfaces. It should grow after incidents, red-team exercises, architecture changes, and new threat intelligence. A prompt library is not a bag of tricks; it is test data for a security control.

Evidence Retention and Closure Testing only matters operationally if evidence survives the exercise. Eval outputs, red-team traces, model versions, prompt templates, retrieved sources, tool-call logs, severity decisions, remediation tickets, and retest results should be stored as security evidence. Closure should require a passing retest, a design change, a compensating control, or explicit risk acceptance. A finding closed because "the team says it is unlikely" is not closure. It is a conversation recorded as a decision.

The Practitioner's Challenge

The political challenge is that red-team findings can embarrass product teams. AI systems often produce strange, vivid, and screenshot-friendly failures. Without agreed severity and scope, stakeholders may argue about whether the finding is "realistic," whether the tester was unfair, or whether the model was merely being creative. The practitioner has to keep the discussion grounded in pre-agreed criteria and production impact.

The structural challenge is that evals often live outside normal release engineering. A model team may run model-quality benchmarks, product engineering may run unit tests, security may run prompt attacks manually, and GRC may ask for evidence separately. If those workflows are disconnected, no one can say whether a model update passed the security suite before release. A useful eval program must connect security testing to CI/CD, change management, and evidence retention.

The technical challenge is writing tests that represent production behavior. Generic jailbreak examples are easy to collect, but production failures often depend on user roles, retrieval content, tool permissions, prompt templates, streaming behavior, and model versions. A system can pass a generic benchmark while failing against the exact workflow customers use. The practitioner must test the system, not just the model.

How to Approach It

Start with the production surfaces. Identify the AI workflows that need evaluation: chat, RAG, summarization, code generation, agent tool use, customer support, internal search, decision support, or external communication. For each surface, define user roles, data sources, model versions, prompt templates, tools, outputs, and release triggers. Do not start from a public benchmark and assume it maps to your product.

Next, define the severity rubric. Write examples for critical, high, medium, low, informational, and out-of-scope findings in your environment. Include data disclosure, unauthorized retrieval, unsafe tool execution, irreversible external action, policy bypass, sensitive output, hallucinated citation, and unsupported claim scenarios where relevant. Make the rubric visible before testing starts. A good rubric gives testers and product teams the same language for impact.

Then build the eval suite around behaviors that should not regress. For each test case, record the surface, scenario, input, required context, expected behavior, severity, regression flag, owner, and release consequence. Some tests should be deterministic pass/fail checks; others may require evaluator judgment. Where model non-determinism matters, run multiple samples and define how failure is counted. The goal is not perfect determinism; it is controlled decision-making.

Run human red-team exercises for discovery. Scope the exercise with model versions, tools, user roles, allowed techniques, exclusions, time box, evidence requirements, and safety boundaries. Encourage testers to explore chains that automated tests do not cover. Require reproduction details rather than just screenshots. At the end, classify findings against the severity rubric and decide which ones become regression tests.

Convert findings into durable controls. A prompt injection finding might become an eval case, a retrieval filter test, a prompt template change, or an output validation rule. An agent misuse finding might become a tool policy constraint, an approval gate, a sandbox limit, and a trace requirement. A citation failure might become a source-support validation test. The conversion step is where red teaming becomes a control rather than an event.

End with evidence and cadence. Decide when evals run: pull request, prompt change, model update, retrieval index change, tool permission change, release candidate, scheduled regression, or after incident remediation. Store outputs in a location that supports audits and customer security reviews. Report trends: failures by severity, time to remediate, recurring classes, release blocks, and open risk acceptances.

Outputs and Deliverables

The core testing artifacts are the eval suite design, prompt attack library, and production surface map. The surface map ties tests to real workflows, user roles, data sources, tool permissions, and model versions. The attack library provides reusable adversarial cases with expected behavior, severity, and reproduction notes. The eval design makes those cases operational by defining execution cadence, pass/fail thresholds, sampling strategy, ownership, and release consequences.

The red-team artifacts are the red-team scope document, severity rubric, and finding classification guide. The scope document prevents argument after delivery by naming included systems, threat actors, allowed techniques, exclusions, time box, and evidence format. The severity rubric establishes impact categories before testing starts. The classification guide helps separate capability limitation, quality failure, safety issue, privacy concern, and security finding so closure follows the right path.

The evidence artifacts are the eval run record, red-team evidence package, closure record, and regression conversion log. Eval run records should include model version, prompt template, system configuration, test case version, outputs, result, and release decision. Red-team evidence packages should preserve prompt, context, retrieved sources, tool calls, outputs, timestamps, screenshots where useful, and tester notes. Closure records should show remediation, retest, exception, or risk acceptance, while the conversion log tracks which findings became permanent tests or controls.

Common Failure Modes

Report Without Regression: The red team delivers findings, but no tests or release gates change afterward. This happens when the exercise is treated as an assessment rather than a control improvement loop. Recover by requiring every valid finding to produce a closure action: regression test, design change, compensating control, or risk acceptance. The report should be the beginning of control improvement, not the end.

Benchmark Substitution: The team uses public benchmarks or model-quality tests as a substitute for production evals. This creates impressive numbers that do not reflect the deployed system's data, tools, prompts, or users. Avoid it by writing tests against real product surfaces and known risk scenarios. Benchmarks can supplement, not replace, production-specific evaluation.

Severity Negotiation: Findings are downgraded after delivery because severity was not defined in advance. This turns closure into politics. Avoid it by agreeing on severity examples before testing begins and applying them consistently. If a finding does not fit the rubric, update the rubric after the exercise, not during the argument.

Evidence Thinness: Findings are captured as screenshots or summaries without reproduction details. Engineering cannot fix confidently and GRC cannot prove closure. Recover by defining evidence requirements before testing: prompt, context, model version, configuration, retrieval sources, tool calls, output, expected behavior, and actual behavior. A finding that cannot be reproduced cannot become a reliable control.

Implementation Checklist

• Map eval coverage to real production surfaces, user roles, data sources, tools, and model versions.
• Define severity categories and examples before running red-team exercises.
• Build a versioned prompt attack library with expected behavior and severity tags.
• Write eval cases that test RAG, agent, prompt, output, and policy behavior separately where possible.
• Configure high-severity eval failures to block release or trigger explicit risk acceptance.
• Require red-team findings to include reproduction evidence, not only screenshots or summaries.
• Convert valid red-team findings into regression tests, control changes, or risk acceptance records.
• Store eval outputs, red-team evidence, closure records, and release decisions as governance evidence.

Related Reading

• Handbook chapters: Chapter 3, Threat Modeling AI Systems; Chapter 4, Prompt Injection and RAG Security; Chapter 5, Agent and Tool-Calling Security; Chapter 8, Governance-to-Engineering Evidence; Chapter 11, Building the Operating Model.
• Field Guide: Red Teaming and Adversarial Evaluations; Prompt Injection and Context Security; RAG Security; Agent Security; Incident Response and AI Observability.

The AI governance program that produces polished documents but cannot answer which systems are in production, who owns each control, and what evidence proves those controls operated last quarter has a policy problem, not a documentation problem. Frameworks like NIST AI RMF, ISO 42001, and OWASP LLM Top 10 describe what mature AI governance looks like. They do not generate the artifacts. That work is engineering, and it requires engineers.

What This Chapter Covers

This chapter covers the translation layer between AI governance language and engineering execution. It explains how framework requirements become inventory fields, threat models, release gates, eval suites, logging requirements, vendor reviews, evidence artifacts, executive reports, and audit-ready packages. The organizational problem it solves is the gap between AI policy and AI control operation: leadership believes governance exists because documents exist, while engineering teams still lack clear owners, tests, artifacts, and release-blocking criteria.

This chapter is relevant when a company adopts an AI policy, maps to NIST AI RMF or ISO 42001, prepares for customer security reviews, responds to board questions, enters a regulated market, or realizes that AI risk language is not connected to product release decisions. It is especially relevant for AI security engineers, GRC leads, security architects, product security teams, and CISO-office practitioners who must turn broad requirements into evidence that a technical team can produce repeatedly.

After working through this chapter, you should be able to build an AI inventory, define control owners, translate framework expectations into engineering artifacts, decide what counts as control evidence, connect governance to release gates, and produce reports that show risk, uncertainty, evidence freshness, and accountability. You should also be able to identify when an AI governance program is operating and when it is merely documented.

Core Concepts

Governance-to-Engineering Translation Frameworks describe intent, but systems require implementation. A governance statement such as "AI systems should be monitored for harmful behavior" must become concrete artifacts: telemetry requirements, detection logic, owner assignment, alert thresholds, review cadence, incident playbook updates, and evidence storage. Translation is the work of converting a policy expectation into a control that operates inside engineering workflows. Without this translation, teams may agree with the policy and still have no idea what to build.

AI Inventory as Foundation Inventory is the first operational governance artifact because you cannot govern what you cannot enumerate. A useful AI inventory includes system ID, owner, business purpose, user population, data categories, model/provider dependencies, retrieval sources, tool access, deployment status, risk tier, vendor involvement, and evidence links. It should connect to procurement, SDLC intake, incident response, and executive reporting. A spreadsheet can start the inventory, but the inventory must become a maintained control, not a one-time survey.

Control Ownership Every AI governance control needs a named owner who can operate it, produce evidence, and respond when it fails. Committees can approve frameworks, but they cannot run retrieval authorization tests or update eval suites. Ownership should be assigned to the team closest to the control: AI engineering for evals, platform for model registry controls, product security for threat models, GRC for evidence cadence, procurement for vendor reviews, and security leadership for risk acceptance. Ambiguous ownership is one of the fastest ways for AI governance to become theater.

Evidence Artifact Taxonomy Not all documents are evidence. A policy describes intent; a training record shows awareness; a risk register records a decision. Control evidence proves that a control operated. Examples include eval gate logs, model intake approvals, retrieval authorization test results, vendor assessment closure records, incident traces, access review records, tool-call audit logs, release gate outcomes, and exception approvals. A governance program needs a taxonomy that separates policy, procedure, evidence, metric, and risk acceptance.

Release Gates as Governance Enforcement Governance becomes real when it changes shipping decisions. If a high-risk AI system lacks a threat model, model approval, eval evidence, retrieval authorization, logging, rollback, or vendor review, the release process should block launch or require explicit risk acceptance. Release gates are how abstract governance requirements become operational boundaries. They also create evidence that the organization did not merely advise teams; it enforced decisions.

The Practitioner's Challenge

The political challenge is that governance often has executive visibility before engineering readiness. Leadership may want a maturity statement, customer-facing assurance language, or board report before the underlying controls exist. Practitioners must tell the truth without sounding obstructive: the organization may have governance intent, but not yet governance evidence. That distinction can be uncomfortable, but it is necessary.

The structural challenge is that evidence lives across many systems. Eval results may live in CI/CD, model approvals in a registry, retrieval logs in observability tooling, vendor reviews in procurement, threat models in security docs, and risk acceptance in GRC tooling. No single team naturally owns the full evidence chain. Governance-to-engineering work requires a control registry that links these artifacts without forcing every team into one tool.

The technical challenge is that AI controls are often new or unstable. Teams may not yet have standardized eval outputs, model intake records, prompt logging policies, or agent tool-call traces. Framework mapping can move faster than implementation. The practitioner must define enough structure to make progress while allowing controls to mature as systems and threats change.

How to Approach It

Start with inventory. Identify all AI systems, features, models, vendors, agents, retrieval indexes, and high-risk workflows in production or planned for production. Record owner, purpose, users, data categories, model dependencies, deployment status, and risk tier. If the inventory is incomplete, say so explicitly. Inventory coverage is itself a governance metric.

Next, map frameworks to control objectives rather than copying framework language into a spreadsheet. For each requirement, ask what system behavior would satisfy it. NIST AI RMF might translate into inventory, threat modeling, evals, monitoring, and risk review. ISO 42001 might translate into management system evidence, ownership, audit cadence, and continual improvement records. OWASP LLM Top 10 might translate into product review tests, release criteria, and red-team coverage.

Then assign owners and evidence. For each control objective, name the operational owner, evidence artifact, collection cadence, storage location, and review process. Avoid committee ownership. If no team can operate the control, the control is not implemented. If no artifact proves operation, the control is not evidenced.

Build release gates around high-risk controls. Not every governance requirement should block every release, but high-risk AI systems need clear launch criteria. Define blockers for missing threat models, failed evals, unapproved model changes, absent retrieval authorization, broad agent permissions, missing logs, or incomplete vendor review. Define who can accept exceptions and for how long.

Create reporting that surfaces uncertainty. Executive reporting should not be a green dashboard that hides weak evidence. Report inventory coverage, evidence freshness, open exceptions, high-risk systems without complete controls, release blocks, eval trends, vendor review gaps, and incident findings. The point is to support decisions, not reassure prematurely.

End by creating a feedback loop. Incidents should update controls. Red-team findings should update evals. Vendor model changes should trigger review. New framework obligations should become backlog items. Evidence gaps should become operating-model work. Governance is not a document cycle; it is a continuous translation loop between obligations, systems, evidence, and decisions.

Outputs and Deliverables

The foundational artifacts are the AI inventory, control registry, and framework translation map. The inventory defines the governed population: systems, owners, data, models, vendors, deployment status, risk tier, and evidence links. The control registry turns governance into accountable operation by listing each control, owner, artifact, cadence, status, last evidence date, and exception state. The framework translation map connects NIST AI RMF, ISO 42001, OWASP LLM Top 10, EU AI Act risk tiers, MITRE ATLAS, and internal policies to the engineering controls that actually satisfy them.

The operating artifacts are the evidence artifact taxonomy, release gate matrix, and risk acceptance record. The taxonomy prevents teams from substituting policy documents for operational evidence by defining what counts as proof for each control type. The release gate matrix specifies which missing or failed controls block launch for each risk tier. The risk acceptance record documents who accepted the risk, why, what compensating controls exist, when the exception expires, and what evidence must be produced before closure.

The assurance artifacts are the AI governance evidence package, executive reporting dashboard, and customer questionnaire response pack. The evidence package is the internal binder that shows inventory, controls, owners, evidence, exceptions, and audit trails. The executive dashboard summarizes posture without hiding uncertainty: coverage, freshness, open gaps, incidents, vendor exposure, and release blocks. The questionnaire pack translates technical evidence into customer-facing language without overclaiming maturity the organization cannot prove.

Framework-to-Evidence Crosswalk

This crosswalk is an engineering evidence map, not legal advice. It uses broad framework themes and maps them to artifacts that help a security team prove control operation. Legal, compliance, and privacy teams should validate jurisdiction-specific obligations before public claims are made.

Synthetic Media and Identity Verification Controls

Synthetic media risk belongs in the handbook because it creates security decisions, not just communications risk. Deepfake-enabled voice calls, synthetic interview candidates, manipulated customer media, forged approval evidence, and generated documents can all enter security workflows. The control question is not whether a team can perfectly detect synthetic content. The control question is whether high-impact decisions rely on media or identity evidence without an independent verification path.

Start by identifying workflows where audio, video, images, or remote identity signals can authorize action or influence trust: executive approvals, payment changes, hiring interviews, customer onboarding, account recovery, fraud review, incident escalation, vendor instructions, and legal or compliance evidence. For each workflow, define which media is advisory, which media is evidence, and which media can trigger action. Anything that can trigger money movement, access changes, employment decisions, customer account changes, or public communications needs stronger controls than human intuition.

Minimum viable controls include out-of-band verification for high-risk approvals, liveness checks for identity proofing, known-channel callback procedures, dual approval for unusual financial or access requests, provenance or watermark review where available, vendor claims review, and incident handling for suspected synthetic media. Human review should be treated as one signal, not the whole control. Reviewers need context, escalation paths, and a clear rule for when media evidence is insufficient.

Evidence artifacts should be lightweight but explicit. A Synthetic Media Verification Record should capture the asset type, workflow, verification method, reviewer, decision, and evidence retained. A Watermark Verification Log can record whether watermark, provenance, or content authenticity signals were checked and what they proved. A Liveness and Identity Verification Review should capture the identity workflow, vendor control, fallback process, false-accept concern, and escalation path. For incidents, the AI Incident Reconstruction Log should record media source, verification steps, decision impact, containment, and follow-up controls.

Do not overclaim detection certainty. Use careful language: the organization applies verification controls, reviews provenance signals where available, requires out-of-band confirmation for high-risk actions, and records evidence for investigation. Avoid claiming that a watermark, detector, or human reviewer proves authenticity by itself.

Common Failure Modes

Policy-First Theater: The organization writes policies before identifying systems, owners, and evidence. The documents look mature, but teams cannot show how controls operate. Recover by building inventory and mapping each policy statement to an artifact and owner. If no artifact exists, the policy is aspiration rather than control.

Framework Spreadsheet Trap: Teams map every framework item to a status column and call the program complete. The spreadsheet may be useful for tracking, but it does not prove operation. Recover by requiring each mapped item to identify the system behavior, control owner, evidence artifact, cadence, and storage location. Framework mapping is not the same as implementation.

Committee Ownership: Controls are assigned to working groups, councils, or governance boards instead of operational teams. This creates meetings without accountability. Recover by assigning each control to a named team that can operate it and produce evidence. Committees can review posture; they should not be the only owners of controls.

Green Dashboard Drift: Executive reporting compresses uncertainty into reassuring status colors. This happens when leaders ask for simplicity and practitioners avoid surfacing gaps. Recover by reporting evidence freshness, inventory coverage, open exceptions, unowned controls, and release blocks alongside status. A useful report helps leaders make decisions, not just feel safe.

Synthetic Approval Trust: A team accepts voice, video, image, or chat evidence as sufficient approval for a high-risk action. This fails when media can be generated, replayed, edited, or impersonated. Recover by requiring known-channel confirmation, liveness or identity checks where appropriate, dual approval for high-risk actions, and a verification record.

Implementation Checklist

• Build an AI inventory with owner, purpose, data categories, model dependency, risk tier, deployment status, and evidence links.
• Translate each governance requirement into a concrete control objective and engineering artifact.
• Assign every control to a named operational owner, not a committee alone.
• Define what counts as evidence for evals, model intake, retrieval authorization, vendor review, incident response, and release gates.
• Create a release gate matrix that blocks high-risk launches when critical evidence is missing.
• Write a risk acceptance record format with owner, rationale, compensating controls, expiration, and closure evidence.
• Define verification controls for media or identity signals that can trigger financial, access, hiring, customer, or public-communication decisions.
• Report inventory coverage, evidence freshness, open exceptions, and unowned controls to leadership.
• Convert audit, incident, vendor, and red-team findings into backlog items and evidence improvements.

Related Reading

• Handbook chapters: Chapter 1, What Is AI Security Engineering?; Chapter 7, Evals, Red Teaming, and Evidence; Chapter 10, Hiring and Assessment; Chapter 11, Building the Operating Model; Chapter 12, Field Kit and Templates.
• Field Guide: AI Governance, Risk, and Compliance; AI-Aware Secure SDLC; Incident Response and AI Observability; Vendor Risk and AI Procurement; Secure AI Architecture Design.

AI security decisions are rarely clean. The eval passes, but the system is being deployed to a context the eval did not cover. The vendor's SOC 2 is current, but their model change notice policy is effectively "we will communicate major updates." The agent's tool permissions look fine in isolation, but no one has analyzed the action chain. Most practitioners who struggle in AI security do not lack technical knowledge; they lack a reasoning pattern for decisions where information is incomplete, model behavior is non-deterministic, and the organization wants certainty that is not available.

What This Chapter Covers

This chapter covers the decision-making habits that separate effective AI security practitioners from technically knowledgeable but operationally limited ones. It explains probabilistic reasoning, risk-tiered decision making, adversarial judgment, systems thinking, uncertainty communication, incident reasoning, ambiguity-aware writing, decision hygiene, and learning cadence. The organizational problem it solves is that AI security work often requires decisions before evidence is complete, before technology stabilizes, and before the organization has a mature control system.

This chapter is relevant when you are reviewing an AI feature under deadline pressure, deciding whether an eval failure should block release, briefing leadership on uncertain risk, classifying a red-team finding, scoping an AI incident, evaluating a vendor's unclear claims, or designing controls for a system that will change after launch. It is also relevant for practitioners moving from deterministic security domains into AI systems where behavior varies across context, model version, prompt template, retrieval state, and user interaction.

After working through this chapter, you should be able to make calibrated AI security judgments without collapsing into false certainty or total paralysis. You should be able to communicate uncertainty clearly, tie recommendations to evidence quality, distinguish unknowns from accepted risks, and reason across layers when failures do not fit one category neatly. You should also be able to recognize your own reasoning errors before they become architecture decisions, severity ratings, or executive narratives.

Core Concepts

Probabilistic Reasoning AI security often deals with likelihood, confidence, and evidence quality rather than binary certainty. A model may usually refuse a class of requests, an eval may pass most cases, and a control may reduce risk without eliminating it. Probabilistic reasoning means stating what you believe, how confident you are, what evidence supports the belief, and what evidence would change your mind. It prevents both overconfidence and blanket pessimism. The practitioner should be comfortable saying, "This is plausible, not proven; here is the decision we can make safely under that uncertainty."

Risk-Tiered Decision Making Not every AI system requires the same rigor. A low-risk internal writing assistant does not need the same evidence as an agent that modifies customer accounts or a RAG system that retrieves regulated data. Risk tiering should account for data sensitivity, user population, action authority, external exposure, reversibility, business criticality, and audit obligation. The operational mindset asks what level of control is proportionate, not whether every theoretical risk has been eliminated. This keeps security credible and focused.

Adversarial Judgment Without Paranoia Adversarial thinking means modeling what a motivated actor would do differently from an ordinary user. It does not mean treating every weird output as an active attack or inventing cinematic threat scenarios disconnected from system design. Useful adversarial judgment identifies realistic preconditions, paths, incentives, and impacts. It asks how an attacker would influence context, retrieval, tools, vendors, or outputs. It then turns those paths into controls, tests, or monitoring.

Systems Thinking Across Layers AI failures often move through layers: context affects model output, model output affects tool arguments, tool output affects a later prompt, and the final result affects a user or workflow. Systems thinking traces the path without getting stuck at the most visible symptom. A hallucinated citation may be a generation problem, citation-binding failure, retrieval issue, or product UX problem. An unsafe tool call may originate in prompt injection, poor runtime authorization, weak approval design, or excessive credential scope. The practitioner follows the chain.

Decision Hygiene Decision hygiene is the discipline of noticing how your own reasoning can fail. Availability bias makes vivid jailbreak examples seem more important than dull retrieval authorization gaps. Confirmation bias makes a red team seek the failure it already expects. Anchoring makes the first severity rating sticky. Approval bias makes human-in-the-loop controls feel stronger than they are. Good practitioners use rubrics, evidence requirements, peer review, and written assumptions to reduce these errors.

The Practitioner's Challenge

The political challenge is that stakeholders often ask for certainty to support a decision they already want to make. Product wants to launch, legal wants defensible language, leadership wants a concise risk statement, and engineering wants clear pass/fail criteria. AI security rarely provides perfect certainty. The practitioner must give decision-useful guidance without pretending the uncertainty is gone.

The structural challenge is that evidence is distributed and uneven. One team may have eval results, another has logs, another knows the model provider contract, another owns the retrieval index, and another understands the tool permissions. A practitioner making an AI security decision often has to reason with partial evidence across organizational boundaries. The work requires not just technical analysis but active evidence gathering and explicit caveats.

The technical challenge is non-determinism and drift. The same system may behave differently after a model update, prompt change, retrieval corpus update, tool integration, or user behavior shift. A one-time test does not prove permanent safety. The practitioner needs to reason in terms of control systems, regression checks, observability, and change triggers rather than one-time approval.

How to Approach It

Start with the decision being made. Are you deciding whether to launch, whether to block release, whether to accept risk, whether to escalate to leadership, whether to classify an incident, or whether to approve a vendor? The same evidence can support different decisions differently. A finding that is acceptable for internal beta may be unacceptable for customer-facing launch. Frame the decision before collecting more facts.

Next, identify the risk tier. Consider data sensitivity, action authority, user population, external exposure, reversibility, regulatory obligation, customer commitment, and business criticality. This determines how much evidence is required and which controls should be mandatory. Risk tiering prevents low-risk features from drowning in process and high-risk systems from slipping through lightweight review.

Then state assumptions and evidence quality. Separate known facts, plausible inferences, open questions, and unsupported claims. A vendor statement is not the same as a log record. A demo is not the same as a regression suite. A model card is not the same as a deployment manifest. Write down the confidence level so the decision does not quietly depend on evidence that is weaker than it appears.

Trace failure paths across layers. Work backward from the bad outcome: unauthorized disclosure, unsafe action, harmful output, audit failure, customer impact, or incident investigation failure. Ask what context, retrieval, model behavior, tool permission, approval, output handling, log, or governance control would have prevented or detected it. This method reveals missing controls better than arguing from abstract risk categories.

Communicate uncertainty in operational language. Avoid both alarmism and reassurance. Say what is known, what is unknown, what could go wrong, what would reduce uncertainty, what control is recommended, and what decision remains with leadership. A good risk statement might say: "We have not proven cross-tenant retrieval isolation. Until the retrieval authorization test passes, this should not launch to multi-tenant production. A single-tenant beta with restricted corpus and additional logging is acceptable."

End by creating a learning loop. If the decision depends on uncertainty, decide what evidence will be collected next and when the decision will be revisited. Turn assumptions into tests, tests into gates, incidents into regressions, and vendor claims into contractual evidence. Operational mindset is not one-time judgment; it is a cadence of calibration.

Outputs and Deliverables

The practical artifacts start with a risk-tiering rubric, decision memo, and assumption log. The rubric defines how data sensitivity, action authority, exposure, reversibility, and evidence quality change the required control level. The decision memo records the choice being made, evidence reviewed, known gaps, recommended controls, residual risk, and decision owner. The assumption log prevents teams from forgetting which parts of the recommendation depended on unproven claims.

The analysis artifacts include an AI failure-path worksheet, evidence quality matrix, and uncertainty register. The failure-path worksheet starts from a bad outcome and traces backward through context, retrieval, model, tool, output, and governance layers. The evidence matrix ranks inputs such as logs, eval results, red-team findings, vendor statements, architecture diagrams, and policy documents by reliability. The uncertainty register records open questions, why they matter, what would resolve them, and what decision can proceed before they are resolved.

The operating artifacts are the risk communication brief, decision hygiene checklist, and learning cadence record. The communication brief gives leadership a clear statement of known risk, uncertainty, options, and recommended next step. The decision hygiene checklist forces reviewers to check for bias, severity drift, missing evidence, and misplaced trust. The learning cadence record tracks which decisions require re-review after model updates, incidents, vendor changes, eval failures, or new threat patterns.

Common Failure Modes

False Precision: The practitioner gives a numeric or categorical answer that the evidence does not support. This happens when stakeholders demand certainty and the practitioner wants to be helpful. Recover by separating confidence from severity and naming the uncertainty explicitly. A precise risk rating based on weak evidence is worse than a qualified recommendation.

Total Paralysis: The team refuses to make any decision because AI behavior is uncertain. This sounds safe but often leads to shadow launches, bypassed review, or loss of credibility. Recover by using risk tiers, scoped approvals, compensating controls, and explicit review dates. The goal is controlled progress, not perfect certainty.

Vivid Attack Bias: A dramatic jailbreak or red-team example dominates prioritization even when a duller control gap is more likely or more damaging. This happens because vivid examples are easier to explain. Recover by comparing failure paths using impact, preconditions, exposure, and evidence. The most memorable risk is not always the highest priority.

Approval Overtrust: The team treats human approval as proof that an action is safe. Approval may be weak if it is frequent, context-poor, or applied after the model has already shaped the decision. Recover by reviewing what the approver actually sees, what alternatives exist, and whether the underlying action should be possible at all. Approval is one control, not a substitute for architecture.

Implementation Checklist

• Define the exact decision being made before collecting or debating evidence.
• Apply a risk-tiering rubric based on data sensitivity, action authority, exposure, reversibility, and evidence quality.
• Separate known facts, plausible inferences, open questions, and unsupported claims in every major review.
• Trace at least one serious failure path across context, retrieval, model, tool, output, and governance layers.
• Write risk statements that include uncertainty, evidence quality, recommended action, and decision owner.
• Check for decision biases such as vivid attack bias, confirmation bias, anchoring, and approval overtrust.
• Convert unresolved uncertainty into tests, logging requirements, vendor questions, or review triggers.
• Revisit decisions after model updates, tool changes, incident findings, vendor changes, or eval failures.

Related Reading

• Handbook chapters: Chapter 1, What Is AI Security Engineering?; Chapter 3, Threat Modeling AI Systems; Chapter 7, Evals, Red Teaming, and Evidence; Chapter 8, Governance-to-Engineering Evidence; Chapter 11, Building the Operating Model.
• Field Guide: AI Security Foundations; Red Teaming and Adversarial Evaluations; Incident Response and AI Observability; Secure AI Architecture Design; Vendor Risk and AI Procurement.

The interview loop that asks "have you done AI security work?" and accepts a confident yes has optimized for self-presentation rather than capability. The candidate who has seen every threat term but built no controls and the candidate who built one eval pipeline extremely well are not equally useful for most roles, but a keyword-based screen treats them identically. Hiring for AI security requires the same rigor the discipline demands everywhere else: specific claims, testable evidence, calibrated evaluation.

What This Chapter Covers

This chapter covers practical hiring and assessment design for AI security roles. It explains archetype-specific interview loops, work samples, scorecards, recruiter enablement, candidate artifact validation, calibration across interviewers, adjacent-background assessment, reference checks, and onboarding signals. The organizational problem it solves is that standard security interview loops often test generic security competence while failing to discriminate between AI security vocabulary, adjacent experience, and real operating capability.

This chapter is relevant when a company writes an AI security req, screens candidates from AppSec, ProductSec, red team, ML, GRC, detection, or architecture backgrounds, or tries to decide whether an internal security engineer can transition into AI security. It is also relevant when hiring managers discover that candidates all mention prompt injection, RAG, agents, evals, and governance but cannot show what they have actually built or reviewed. The chapter is written for hiring managers, recruiters, interviewers, security leaders, and practitioners who want their own experience to be evaluated accurately.

After working through this chapter, you should be able to design an interview loop for the nine AI security archetypes, write practical exercises that test real judgment, build a scorecard that does not require perfection across every domain, validate claims through artifacts, and onboard a first AI security hire into a team that has not done the work before. You should also be able to identify resume red flags without dismissing strong adjacent candidates who have the right reasoning pattern.

Core Concepts

Archetype-Specific Hiring AI security is not one role shape. An AI Security Architect, AI Product Security Engineer, AI AppSec Engineer, RAG Security Engineer, Agent Security Engineer, AI Red Team Engineer, ML Security Engineer, Model Risk Security Partner, and Governance Evidence Lead require different evidence and interview design. A candidate who is excellent at RAG threat modeling may not be the right first hire for governance evidence. A candidate who can design control registries may not be the person to run prompt injection testing. The hiring loop should test the archetype the organization needs, not a generic AI security fantasy.

Work Samples Over Vocabulary AI security vocabulary is easy to learn at the surface level. Work samples reveal reasoning. A RAG threat model exercise, tool permission review, model intake critique, eval design prompt, governance evidence mapping task, or architecture diagram review shows how the candidate thinks under realistic constraints. The goal is not to create a burdensome unpaid project. The goal is to test the same judgment the job requires.

Artifact Validation Claims should be tied to artifacts where possible. If a candidate says they ran an AI red team, ask about scope, severity rubric, evidence format, closure criteria, and which findings became regression tests. If they built evals, ask how failures blocked release and how the suite handled model non-determinism. If they designed RAG authorization, ask what metadata survived chunking and how deletion propagation was tested. Real experience leaves operational residue.

Scorecard Calibration A scorecard should weight the role's core competencies and distinguish required depth from adjacent awareness. For an Agent Security Engineer, tool authorization, blast-radius reasoning, approval design, and audit trails matter more than deep model training knowledge. For a Governance Evidence Lead, framework translation, evidence taxonomy, control ownership, and executive reporting matter more than writing jailbreak prompts. Calibration prevents interviewers from over-weighting their own specialty.

Adjacent Background Translation Strong AI security candidates may come from AppSec, ProductSec, red teaming, detection engineering, GRC, ML platform, privacy, or security architecture. Adjacent backgrounds translate when the candidate can reason across AI-specific layers and produce relevant artifacts. AppSec translates well into AI AppSec when the candidate understands context, retrieval, and model output handling. GRC translates into governance evidence when the candidate can turn frameworks into engineering artifacts instead of policy decks.

The Practitioner's Challenge

The political challenge is that AI security hiring often happens under anxiety. Leaders want confidence that the organization is addressing AI risk, recruiters want searchable keywords, and hiring managers want someone who can cover the whole field. This pressure produces inflated reqs and weak assessment loops. The practitioner designing the process must narrow the role without making it seem less strategic.

The structural challenge is interviewer capability. Many organizations do not yet have enough internal AI security depth to evaluate candidates consistently. Interviewers may ask trivia, over-focus on jailbreaks, or treat experience with GPT as meaningful evidence. Calibration requires prepared rubrics, practical exercises, and interviewer guidance. Otherwise, the process rewards confidence and vocabulary over judgment.

The organizational challenge is onboarding. A strong hire can fail if they arrive into a team with no AI inventory, no clear ownership, no release touchpoints, no executive mandate, and no access to product decisions. Hiring is not the end of role design. The first 30/60/90 days must connect the hire to systems, stakeholders, decisions, and artifacts quickly enough to avoid becoming a reactive help desk.

How to Approach It

Start by choosing the archetype. Use the role architecture from Chapter 2 to decide which of the nine archetypes the organization needs now. Do not write a req until you know whether the hire is primarily reviewing AI product features, building evals, designing agent controls, mapping governance evidence, securing RAG, securing model supply chain, supporting model risk, or setting architecture. If the role needs broad coverage, name the primary archetype and two adjacent areas rather than all nine.

Next, translate the archetype into outcomes. Write the responsibilities as artifacts and decisions: "produce RAG threat models," "define tool permission matrices," "build eval release gates," "map framework controls to evidence," "write model intake records," or "design secure AI reference architectures." This attracts candidates who understand the work and filters out people who only match terms. It also gives interviewers something concrete to test.

Then design the interview loop around evidence. A recruiter screen should test for relevant domain exposure and artifacts, not deep technical proof. The hiring manager interview should validate scope, judgment, and role fit. Technical interviews should use scenario exercises tied to the archetype. Cross-functional interviews should test communication with product, engineering, GRC, or leadership. Every interview should have a purpose.

Build practical exercises that are short, realistic, and reviewable. For AI Product Security, give a feature launch plan and ask for security release blockers. For AI AppSec, give an LLM application flow and ask for threat model findings. For RAG Security, give a retrieval architecture and ask where authorization, chunk metadata, and tenant isolation can fail. For AI Red Team, ask for a scoped eval plan and severity rubric. For Agent Security, ask for a tool permission matrix and approval design. For Governance Evidence, ask the candidate to translate a governance requirement into controls and evidence. For ML Security, ask for a model intake and provenance critique. For Model Risk Security Partner, ask how security evidence should support a model-risk decision. For AI Security Architect, ask for trust-boundary review across a multi-component system.

Use scorecards that separate depth, breadth, judgment, communication, and operating maturity. Do not penalize a candidate for lacking depth in a domain the role does not own. Do penalize vague claims, inability to reason from mechanism, and absence of artifact thinking. Include a field for evidence quality: did the candidate describe work they personally performed, a team they participated in, or concepts they only read about?

End by designing onboarding before the offer is accepted. Define the first systems the hire will review, the stakeholders they will meet, the artifacts they will produce, and the decisions they will influence in 30, 60, and 90 days. If the organization cannot name those, the role is not ready. A good candidate will notice.

Outputs and Deliverables

The hiring foundation includes the archetype-specific job description, role outcome map, and candidate evidence profile. The job description states the primary archetype, adjacent coverage areas, responsibilities, artifacts, non-responsibilities, and operating context. The role outcome map connects the hire to decisions such as release reviews, red-team planning, governance evidence, model intake, or architecture approval. The candidate evidence profile defines what credible experience looks like for the role: threat models, eval suites, tool matrices, registry controls, governance maps, incident traces, or architecture decision records.

The interview system includes the interview loop plan, practical work sample, and scorecard rubric. The loop plan assigns each interviewer a specific signal to test so candidates are not asked the same generic questions repeatedly. The work sample gives candidates a realistic but bounded scenario that tests judgment without requiring excessive unpaid labor. The scorecard weights the role's core competencies, adjacent awareness, evidence quality, communication, and operating maturity.

The enablement and onboarding artifacts include the recruiter screen guide, artifact validation question bank, reference check guide, and 30/60/90-day onboarding plan. The recruiter guide helps screen for actual AI security work rather than AI enthusiasm. The validation question bank gives interviewers follow-up questions for claims such as "I ran an AI red team" or "I built RAG security controls." The onboarding plan connects the new hire to inventory, top-risk systems, stakeholders, first deliverables, and the first operating review.

Common Failure Modes

Frankenstein Req: The job description asks for all nine archetypes with equal depth. This happens when leaders want one person to solve every AI security concern. Recover by naming the primary archetype, adjacent coverage, and explicit non-responsibilities. A narrower role is not less strategic if it is tied to real outcomes.

Jailbreak Interview Bias: Interviewers over-weight prompt injection tricks and under-test retrieval, agents, evidence, supply chain, or operating judgment. This happens because jailbreak examples are easy to ask about. Recover by using scenario exercises that match the role and by testing control design, not just attack familiarity. AI security is broader than prompt cleverness.

Artifact-Free Claim Acceptance: The team accepts claims such as "I built evals" or "I worked on AI governance" without probing for concrete artifacts. This rewards confidence over experience. Recover by asking for scope, owners, outputs, failure cases, evidence, and how the work affected release decisions. Real work has shape.

No Landing Zone: The hire starts without inventory, stakeholder access, release touchpoints, or clear first deliverables. They become reactive and lose influence. Recover by preparing a 30/60/90-day plan before the start date. A first AI security hire needs organizational scaffolding, not just a laptop and a backlog.

Implementation Checklist

• Choose the primary AI security archetype before writing the job description.
• Define adjacent coverage areas and explicit non-responsibilities for the role.
• Write responsibilities as artifacts and decisions, not trend keywords.
• Build a practical exercise matched to the archetype being hired.
• Create a scorecard that weights core depth, adjacent awareness, evidence quality, communication, and operating maturity.
• Train recruiters to screen for artifacts, systems, and control work rather than AI vocabulary alone.
• Prepare follow-up questions that validate claims about evals, red teams, RAG authorization, agent controls, model intake, and governance evidence.
• Build a 30/60/90-day onboarding plan tied to real systems, stakeholders, and first deliverables.

Related Reading

• Handbook chapters: Chapter 2, Role Architecture and Team Design; Chapter 9, The Operational Mindset; Chapter 11, Building the Operating Model; Chapter 12, Field Kit and Templates.
• Field Guide: AI Security Foundations; AI-Aware Secure SDLC; Red Teaming and Adversarial Evaluations; AI Governance, Risk, and Compliance; Secure AI Architecture Design.

An AI security operating model is the difference between a practitioner who responds to whatever arrives and a function that produces consistent controls, evidence, and decisions. Most organizations reach the practitioner stage first: someone is reviewing AI features, answering vendor questions, reacting to incidents, and helping product teams reason through risk. Fewer reach the function stage, where that work is systematic, measured, owned, and accountable to a cadence. The operating model turns individual effort into institutional capability.

What This Chapter Covers

This chapter covers how to run AI security as a repeating operational discipline rather than a sequence of ad hoc reviews. It explains operating cadence, capability ownership, control registries, release gate integration, vendor review, model intake, evidence collection, red-team scheduling, metrics, escalation paths, reporting, maturity progression, and continuous improvement. The organizational problem it solves is that AI security work often starts as expert judgment and stays there too long, leaving the organization dependent on a small number of people instead of a reliable system.

This chapter is relevant when an organization has more AI work than one person can handle reactively. The trigger may be multiple product teams shipping AI features, customer security questionnaires asking for AI evidence, governance frameworks entering the business, agentic workflows reaching production, or a CISO asking for quarterly AI risk reporting. It is also relevant when existing AppSec, ProductSec, GRC, privacy, procurement, and ML platform teams all touch AI risk but no one can explain how the pieces fit together.

After working through this chapter, you should be able to design an AI security operating cadence, assign control ownership, build a control registry, define release gates, choose metrics that reflect posture rather than activity, create escalation paths, and run a quarterly operating review. You should also be able to describe the maturity progression from reactive support to systematic evidence production and continuous improvement.

Core Concepts

Operating Cadence An operating cadence defines the recurring activities that make AI security reliable. Weekly activities may include intake review, release review, high-risk design review, and remediation follow-up. Monthly activities may include evidence collection, vendor review status, eval trend review, and control gap review. Quarterly activities may include red-team planning, executive reporting, risk acceptance review, control refresh, and roadmap updates. Cadence prevents AI security from becoming a pile of urgent requests with no learning loop.

Capability Ownership AI security capability areas need owners, even when execution spans teams. Someone must own AI application review, RAG security, agent controls, model supply chain, evals, observability, vendor AI risk, and governance evidence. Ownership does not mean one person does all the work. It means a named team is accountable for the control operating, evidence existing, and gaps being escalated. Without ownership, AI security becomes coordination theater.

Control Registry A control registry is the operational memory of the AI security function. It lists controls, owners, affected systems, evidence requirements, collection cadence, current status, last verification date, exceptions, and related risks. The registry should not be a static compliance artifact. It should drive reviews, reporting, release gates, and remediation. A control registry lets the organization answer: which controls exist, where do they apply, are they current, and who is accountable?

Release Gate Integration AI security becomes durable when it influences shipping decisions. Release gates define what must be true before AI systems launch or change: threat model completed, model approved, evals passed, retrieval authorization tested, agent tool permissions reviewed, observability in place, rollback planned, vendor review complete, and risk accepted where needed. Gates should be risk-tiered. Low-risk internal features may need lightweight checks; high-risk systems need formal blockers and evidence.

Continuous Improvement Loop An operating model must learn. Red-team findings should become evals. Incidents should become release gates or logging requirements. Vendor model changes should trigger re-review. New threat patterns should update review checklists. Control failures should update ownership, training, and tooling. Continuous improvement is how the function avoids repeating the same AI security lesson every quarter.

The Practitioner's Challenge

The political challenge is that operating models can sound bureaucratic to teams trying to ship. Product teams may support security in principle while resisting additional gates, forms, meetings, or evidence requests. The practitioner must demonstrate that the operating model reduces surprise and accelerates good decisions. A well-designed model should create predictable paths, not random friction.

The structural challenge is that AI security work crosses existing functions. AppSec may own secure SDLC, ProductSec may own feature review, ML platform may own registries and deployment, GRC may own evidence, privacy may own data rights, procurement may own vendor diligence, and legal may own regulatory obligations. The operating model must define interfaces between these teams. If it does not, AI security will either duplicate existing work or fall through gaps.

The measurement challenge is that easy metrics are often misleading. Counting AI policies, training completions, review meetings, or total eval cases may show activity without showing risk reduction. The metrics that matter are harder: evidence freshness, release blocks triggered, high-risk systems without complete controls, mean time to triage AI incidents, vendor review completion, eval failure trends, open risk acceptances, and unowned controls. The function must measure posture, not performance theater.

How to Approach It

Start with the capability map. List the AI security capability areas your organization needs: AI application review, prompt and context security, RAG security, agent controls, model supply chain, MLOps platform security, evals and red teaming, incident observability, vendor risk, privacy support, governance evidence, and secure architecture. For each area, name the operational owner, supporting teams, current maturity, and evidence gap. This gives the operating model a real surface.

Next, define intake and risk tiering. Every AI system or material AI change should enter through an intake path that captures owner, purpose, data category, model dependency, retrieval sources, tool access, vendor involvement, deployment status, and user population. Use those fields to assign a risk tier. The tier determines which reviews, gates, evidence, and approvals apply. This prevents every AI feature from receiving the same process.

Then build the control registry. Translate the capability map into controls with owners, evidence artifacts, cadence, and status. Examples include model intake approval, retrieval authorization testing, prompt injection evals, agent tool permission review, vendor AI addendum, prompt logging policy, incident trace schema, and release gate outcomes. Keep the registry close to operating workflows. If it is updated only before audits, it is not operating.

Integrate with release and change management. Define which AI changes trigger review: new model, model version change, prompt template change, retrieval corpus change, new tool permission, new vendor route, new high-risk use case, or major UI/output behavior change. Map each trigger to required checks. Build the path into CI/CD, product launch review, architecture review, procurement, or model registry promotion rather than creating a separate disconnected approval universe.

Create escalation and risk acceptance paths. Decide which findings can be resolved at team level, which require security leadership, which require CISO approval, and which require executive or legal visibility. Define what a risk acceptance record must contain: owner, rationale, affected systems, compensating controls, expiration, and closure evidence. Without this, unresolved AI risk becomes accepted by silence.

End with reporting and review cadence. Weekly reviews should manage intake and blockers. Monthly reviews should examine evidence freshness, open gaps, vendor changes, incidents, and eval trends. Quarterly reviews should assess maturity, resource needs, major risk acceptances, roadmap progress, and board-level reporting. The cadence should produce decisions, not just status.

Outputs and Deliverables

The foundational operating artifacts are the AI security capability map, AI intake workflow, and risk-tiering model. The capability map shows which work exists, who owns it, what evidence it produces, and where maturity is weak. The intake workflow makes sure new AI systems, model changes, retrieval changes, tool additions, and vendor AI features become visible before launch. The risk-tiering model keeps process proportional by tying review depth to data sensitivity, action authority, user population, exposure, reversibility, and regulatory relevance.

The control and decision artifacts are the control registry, release gate matrix, and risk acceptance process. The control registry gives the function operational memory: control owner, affected systems, evidence type, cadence, status, gaps, and last verified date. The release gate matrix defines what blocks launch at each risk tier and what evidence resolves the blocker. The risk acceptance process makes exceptions explicit, time-bound, owner-backed, and reviewable rather than letting risk disappear into project pressure.

The management artifacts are the operating cadence calendar, metrics dashboard, executive reporting pack, and quarterly operating review agenda. The cadence calendar defines weekly, monthly, and quarterly activities with owners and outputs. The metrics dashboard tracks posture signals such as eval pass rate, release blocks, evidence freshness, incident triage time, vendor review completion, open risk acceptances, unowned controls, and high-risk systems without full coverage. The executive pack translates those signals into decisions about investment, staffing, risk acceptance, and roadmap priority.

Operating Case Studies

Case Study 1: RAG Authorization Failure

Scenario: A support assistant retrieves semantically relevant customer documents across tenant boundaries. Control failure: Similarity ranking runs before tenant, role, account, and document authorization filters. Impact: Unauthorized customer context can enter prompts even if the final answer does not quote it directly. Correct control: Apply retrieval-time authorization before ranking, fail closed when metadata is missing, and log selected chunk IDs. Evidence artifact: Retrieval Authorization Test Record. Postmortem question: Which source, chunk, metadata, and authorization decision allowed the wrong document into context?

Case Study 2: Indirect Prompt Injection Through Retrieved Content

Scenario: A hostile instruction is embedded in a ticket, document, or imported web page that later becomes retrieved context. Control failure: The model treats retrieved content as instruction-bearing context instead of evidence to summarize or cite. Impact: The assistant changes output, suppresses warnings, fabricates authority, or attempts unsafe tool use based on untrusted content. Correct control: Label context trust tiers, separate content from instructions, constrain tool access, and add indirect injection regression tests. Evidence artifact: Prompt Injection Test Record. Postmortem question: Which context source was allowed to influence policy, tool behavior, or system instructions?

Case Study 3: Agent Overbroad Tool Access

Scenario: An agent with broad CRM or ticketing permissions sends, edits, closes, or deletes records based on a malicious or mistaken instruction. Control failure: The tool credential can do more than the workflow requires, and approvals do not show enough context. Impact: A single confused or compromised workflow can create customer-visible errors, data exposure, or business-process damage. Correct control: Scope tools by action class, resource, tenant, and reversibility; require meaningful approval for high-risk actions. Evidence artifact: Agent Blast-Radius Worksheet and Tool Permission Matrix. Postmortem question: What was the maximum action the credential could perform, independent of the tool description?

Case Study 4: Unsafe Model Artifact Loading

Scenario: A team downloads a model artifact, adapter, tokenizer, or helper package from an untrusted or weakly reviewed source and loads it in a production-adjacent environment. Control failure: There is no provenance record, hash verification, unsafe serialization policy, license review, or registry promotion gate. Impact: The environment may execute unsafe loading code, deploy unreviewed behavior, or lose the ability to prove which artifact ran. Correct control: Require model intake, artifact integrity checks, source review, approved loading formats, and registry-based promotion. Evidence artifact: Model Intake Record and Model Provenance Record. Postmortem question: Could the team prove artifact source, hash, loader behavior, license, approval, and deployment target?

Case Study 5: Governance Without Evidence

Scenario: The organization has an AI policy and executive dashboard, but no release gate, owner, log, or artifact proving the policy operated. Control failure: Governance language is disconnected from engineering controls and product-release decisions. Impact: Leaders believe the risk is managed while teams ship AI systems without test evidence, owner records, or exception handling. Correct control: Map each policy requirement to a control owner, evidence artifact, cadence, release gate, and risk acceptance path. Evidence artifact: Governance Evidence Map and Board-to-Backlog Traceability Record. Postmortem question: Which policy statement changed an engineering decision, and what artifact proves it?

Common Failure Modes

Reactive Expert Trap: The organization relies on one expert to answer every AI security question. This works temporarily but does not scale, and it creates inconsistent decisions when the expert is absent or overloaded. Recover by turning repeated expert judgments into checklists, gates, templates, and control ownership. The goal is not to remove expert judgment; it is to reserve it for genuinely hard cases.

Activity Metrics Theater: The function reports number of reviews, number of policies, number of eval cases, or number of meetings as posture. These metrics can hide that high-risk systems still lack evidence or ownership. Recover by measuring evidence freshness, release blocks, control coverage, incident response readiness, vendor gaps, and open exceptions. Activity matters only when it changes risk.

Disconnected Governance: GRC maps frameworks while engineering runs separate reviews and product teams ship through separate release paths. Everyone is busy, but the outputs do not connect. Recover by linking framework controls to release gates, evidence artifacts, and operational owners. Governance must ride the same rails as engineering decisions.

Unowned Control Drift: A control exists in a document but no team maintains it after launch. Over time, model versions change, prompts change, retrieval indexes change, vendors change, and the control becomes stale. Recover by assigning owners, collection cadence, and re-verification triggers. Controls need maintenance like software.

Implementation Checklist

• Build an AI security capability map with owners, controls, evidence, and current maturity.
• Define an intake workflow for new AI systems and material AI changes.
• Create a risk-tiering model based on data sensitivity, action authority, exposure, user population, reversibility, and regulatory relevance.
• Build a control registry with owner, evidence artifact, cadence, status, gap, exception, and last verified date.
• Define AI-specific release gates and change triggers for models, prompts, retrieval, tools, vendors, and high-risk use cases.
• Establish risk acceptance paths with required owner, rationale, compensating controls, expiration, and closure evidence.
• Track posture metrics such as evidence freshness, eval failure rate, release blocks, incident triage time, vendor completion, and unowned controls.
• Run a quarterly AI security operating review with decisions, not just status updates.

Related Reading

• Handbook chapters: Chapter 1, What Is AI Security Engineering?; Chapter 2, Role Architecture and Team Design; Chapter 8, Governance-to-Engineering Evidence; Chapter 9, The Operational Mindset; Chapter 12, Field Kit and Templates.
• Field Guide: AI Governance, Risk, and Compliance; AI-Aware Secure SDLC; Incident Response and AI Observability; Vendor Risk and AI Procurement; Secure AI Architecture Design.

The templates in this chapter are not polish. They exist because AI security fails when teams cannot operationalize the words they use. A control that does not produce evidence is a claim. A policy that does not affect a release decision is advice. A red team that does not produce closure criteria is theater. A hiring req that describes all nine archetypes is a unicorn hunt.

These artifacts are the executable version of the thinking in the previous eleven chapters. Copy them, adapt them, and deploy them without waiting for a mature program to appear first. They assume a roughly 500-person organization with active AI product development, a small security team, a product engineering function, some GRC responsibility, and a need to answer customer or executive questions with evidence.

1. AI Security Scope Statement

Example

AI Security Engineering owns the security review, control design, evidence requirements, and operating model for AI-enabled systems that process company, customer, employee, or regulated data; influence user-facing outputs; retrieve internal or customer content; call tools; automate decisions; or depend on model artifacts, model providers, or AI-specific vendors.

The function is responsible for AI application security, prompt and context security, RAG and retrieval-plane security, agent and tool-use security, model supply chain review, AI-aware SDLC gates, AI red-team and eval evidence, AI observability requirements, and governance-to-engineering evidence. The function partners with AppSec, ProductSec, ML platform, privacy, GRC, legal, procurement, infrastructure, and product engineering. It does not independently own broad AI ethics strategy, employment policy, product-market decisions, legal interpretation, or general corporate AI strategy, though it provides technical evidence and risk analysis for those decisions.

AI Security Engineering's core output is not policy language alone. Its output is enforceable controls, release decisions, review artifacts, test evidence, threat models, model intake records, retrieval authorization evidence, tool permission designs, incident traces, vendor AI assessments, and executive-ready risk summaries. Where controls cannot yet be implemented, the function records risk acceptance with owner, rationale, compensating controls, expiration, and required closure evidence.

Adaptation note

Use this statement as the opening definition for an internal AI security charter. Replace the partner functions with your actual teams. If your organization is smaller, collapse ownership into fewer roles but keep the boundary language. If your organization is regulated, add explicit references to audit readiness, customer assurance, and evidence retention.

2. AI Security Capability Map

Example

Adaptation note

This grid should become a living operating artifact. Review it monthly until the program stabilizes, then quarterly. The maturity labels should be honest and evidence-based. A capability is not Level 2 because a policy exists; it is Level 2 when a repeatable process produces artifacts on a cadence.

3. AI Threat Model Template

Example

System Walkthrough

System name: Customer Support RAG Assistant Business purpose: Help support agents answer customer questions using internal documentation, prior tickets, and account-specific knowledge. Primary users: Support agents and support managers. User-visible output: Suggested answers, citations, escalation recommendations. Downstream effects: Agent may copy response into customer email; assistant does not send directly. Model dependency: Hosted LLM provider through server-side API proxy. Retrieval sources: Product docs, support playbooks, prior tickets, account notes. Sensitive data: Customer account data, support ticket history, internal escalation notes. Risk tier: High because the system retrieves customer data and influences external communications.

Boundary Map

Layered Surface Inventory

Risk Rubric

Critical findings include cross-customer retrieval, unauthorized exposure of account data, or assistant behavior that sends or prepares externally visible false customer commitments. High findings include repeatable indirect injection that changes answer content, missing retrieval audit logs, or citation failures in customer-impacting workflows. Medium findings include weak output validation, incomplete source metadata, or non-blocking eval gaps. Low findings include wording issues, unclear UI warnings, or isolated unsupported claims with no sensitive data.

Release-Blocker List

The feature may not launch until retrieval-time authorization tests pass for cross-customer and cross-role access, prompt injection tests cover retrieved tickets and documentation, model provider retention has been reviewed, citation binding is implemented, and logs include user, account, retrieved source IDs, model version, prompt template version, and output ID. If any of these are missing, the CISO or delegated risk owner must sign time-bound risk acceptance.

Evidence Plan

Store the threat model, retrieval authorization test results, indirect injection test results, vendor review, prompt template version, citation validation report, and logging schema in the AI evidence repository. Link these records from the AI inventory entry for the system. Re-run retrieval and injection tests after changes to source systems, chunking, embedding model, prompt template, model provider, or authorization logic.

Adaptation note

Use the same structure for agents, copilots, coding assistants, internal search, or decision-support systems. Replace the layers with the ones that matter for the system under review. The template should always end with release blockers and evidence, not just findings.

4. RAG Security Checklist

Example

Ingestion

• Each source corpus has an owner, data classification, permission model, update cadence, and deletion behavior.
• The ingestion pipeline preserves source ID, tenant, document owner, ACL reference, classification, version, and ingestion timestamp on every chunk.
• User-generated or low-review sources are labeled as data-safe only, not instruction-safe.
• Ingestion rejects documents whose metadata cannot be mapped to retrieval policy.

Authorization

• Retrieval applies tenant, user, role, document, classification, and purpose filters before similarity ranking.
• The retrieval service fails closed when required identity or authorization metadata is missing.
• Authorization tests prove users cannot retrieve chunks from other tenants, accounts, roles, or classification zones.
• Permission changes in source systems propagate to retrieval eligibility.

Tenancy

• The vector-store tenancy model is documented as shared index, tenant namespace, separate index, or separate store.
• Shared indexes require mandatory metadata filters enforced by service code, not UI convention.
• High-sensitivity data uses stronger isolation or explicit risk acceptance.
• Cache keys include tenant, user or role scope, corpus, model version, and authorization state where relevant.

Metadata

• Chunk metadata includes source ID, chunk ID, tenant, classification, ACL reference, version, ingestion time, and deletion status.
• Metadata cannot be modified by ordinary users through document content.
• Retrieval logs include selected chunk IDs and metadata filters.
• Source-to-chunk lineage is queryable during incident response.

Citation

• Citations bind to retrieved chunk IDs, not model-generated source names.
• Answers that cite sources can be traced to chunks that actually support the claim.
• Citation validation tests cover unsupported claims, stale sources, and wrong-document attribution.
• User-facing UI distinguishes retrieved evidence from generated synthesis.

Deletion Propagation

• Source deletion removes or invalidates chunks, embeddings, cached retrieval results, and generated summaries where required.
• Deletion propagation has a test record with source ID, deletion time, index update, and verification query.
• Re-indexing jobs preserve deletion and permission state.
• Privacy or legal hold exceptions are recorded explicitly.

Adaptation note

Use this checklist during design review and again before launch. Do not collapse authorization and prompt injection into one test. A RAG system can be injection-resistant and still retrieve unauthorized data, or authorization-correct and still follow malicious retrieved instructions.

5. Agent Blast-Radius Worksheet

Example

Required Design Questions

• What credential does the tool actually use?
• Can the credential perform actions broader than the tool description?
• What user, tenant, and resource constraints are enforced at runtime?
• Can one tool call cause external, irreversible, destructive, or privilege-changing effects?
• Can multiple low-risk calls compose into a high-risk chain?
• What does the approval screen show?
• What logs prove who requested, authorized, approved, and executed the action?
• What rollback path exists, and what actions cannot be fully rolled back?

Adaptation note

Use this worksheet before connecting tools to an agent. If the worksheet is filled out after launch, it will mostly document risks that are already live. For high-risk tools, require engineering signoff before implementation and security signoff before production enablement.

6. Model Intake Checklist

Example

Identity and Source

• Model name, version, source URL, publisher, and retrieval date are recorded.
• Artifact hash is calculated and stored before review.
• Artifact is mirrored to controlled internal storage after approval.
• Production deployment uses internal pinned artifact, not public latest or branch reference.

Provenance and Lineage

• Base model is identified and approved.
• Fine-tune, adapter, tokenizer, embedding model, or preprocessing dependencies are documented.
• Training or fine-tuning data categories are recorded where known.
• Known limitations and intended use are documented.

Format and Loading

• Artifact format is classified as allowed, restricted, sandbox-only, or prohibited.
• Pickle or custom-code loaders require sandboxing or exception approval.
• Safetensors or safer formats are preferred where available.
• Loader code is reviewed when model loading executes repository code.

License and Use

• License permits intended commercial or internal use.
• Attribution, redistribution, field-of-use, and output restrictions are recorded.
• Fine-tune inherits base model license obligations where applicable.
• Legal review is completed for customer-facing or commercial deployment.

Eval and Security Evidence

• Required evals passed for intended use.
• Red-team or abuse testing completed for high-risk deployments.
• Model card or internal limitations record is linked.
• Rollback version is identified before production promotion.

Promotion Approval

• Owner approves production use.
• Security approves supply-chain review.
• Legal or procurement approves license and provider terms where required.
• Registry entry includes metadata, evidence links, approval state, and deployment target.

Adaptation note

Use this checklist for model weights, adapters, embedding models, rerankers, tokenizers, and preprocessing artifacts that influence production behavior. For hosted model APIs, adapt the checklist into a provider and model-version intake record.

7. Red-Team Scope Document

Example

Exercise name: Customer Support RAG Assistant Red Team System under test: Support assistant in staging environment with production-like documents and synthetic customer accounts. Model versions: Hosted model provider version 2026-02-stable, prompt template support-rag-v4, retrieval service retriever-2.1. User roles: Support agent, support manager, unauthorized support contractor. Threat actors: Malicious customer, compromised internal user, support agent attempting unauthorized access, external attacker influencing imported documents. Allowed techniques: Direct prompt injection, indirect injection through uploaded documents and tickets, retrieval poisoning in approved test corpus, citation manipulation, policy bypass attempts, cross-role access attempts. Excluded techniques: Live customer data access, production email sending, destructive ticket actions, provider account attacks, denial-of-service testing. Time box: Five business days of testing plus two days for reproduction and evidence packaging. Evidence format: Each finding must include prompt, context source, retrieved chunk IDs, model version, prompt template version, output, expected behavior, actual behavior, severity, reproduction steps, and recommended control. Severity rubric: Critical means unauthorized cross-customer data exposure, irreversible external action, or repeatable bypass affecting production release criteria. High means unauthorized role-level access, reliable indirect injection affecting answer integrity, or missing logs for high-risk workflow. Medium means constrained bypass with limited impact or missing evidence that weakens assurance. Low means weak UX or policy ambiguity without immediate security property violation. Informational means behavior worth tracking but not a control failure. Closure criteria: Critical and high findings require retest evidence or formal risk acceptance. Medium findings require owner assignment and remediation plan. Low and informational findings require product/security triage. Reporting audience: Product engineering, product security, AI security, support leadership, GRC.

Adaptation note

Do not start a red team without this scope. If the assessed team has not agreed to severity and closure criteria, you are setting up a political argument instead of an evaluation.

8. Eval Design Template

Example

Required Fields

Each eval case should include owner, model version, prompt template version, dataset version, execution date, result, failure evidence, and release consequence. For non-deterministic outputs, define sampling count and failure threshold. For high-risk cases, one failure may be enough to block release.

Adaptation note

Treat evals as release controls, not quality demos. Generic prompt tests are useful only if they map to a production surface or known failure class. Every critical or high red-team finding should be evaluated for conversion into this format.

9. Governance Evidence Scorecard

Example

Adaptation note

Use this scorecard in monthly reviews. "Last verified" should reflect evidence freshness, not the date someone updated the spreadsheet. Risk acceptance should be time-bound and owned.

10. AI Vendor AI-Addendum Checklist

Example

Model and Provider

• Vendor identifies model provider, model family, deployment mode, and whether customer-specific fine-tuning is used.
• Vendor provides model change notice terms for material model, provider, or routing changes.
• Vendor explains whether customers can disable, defer, or test model changes before rollout.
• Vendor states whether the feature uses retrieval, embeddings, agents, or automated decisions.

Customer Data

• Vendor states whether prompts, uploads, files, tickets, feedback, or outputs are used for training, fine-tuning, evals, abuse monitoring, or product improvement.
• Vendor provides opt-out terms and evidence of tenant isolation.
• Vendor identifies retention periods for prompts, outputs, retrieved context, and logs.
• Vendor identifies human review conditions and reviewer access controls.

Output Rights and Auditability

• Contract states who owns generated outputs.
• Contract identifies sublicensing, attribution, watermarking, or disclosure obligations.
• Vendor explains what logs are available after an AI-generated error or harmful decision.
• Vendor provides audit rights or incident support terms for AI-generated outputs.

Security and Governance

• Vendor provides AI security testing summary or eval evidence for relevant features.
• Vendor discloses agent tool permissions or external actions if applicable.
• Vendor identifies AI subprocessors and data locations.
• Vendor agrees to notify customer of AI incidents affecting customer data, outputs, or decisions.

Adaptation note

Add this to existing vendor security review rather than replacing the standard questionnaire. AI review supplements infrastructure review; it does not make SSO, encryption, vulnerability management, and incident response irrelevant.

11. Named Evidence Artifact Templates

Use these compact templates as the minimum field kit for recurring AI security evidence. Each template should live where the owning team can update it and where GRC, incident response, and security leadership can find it during reviews.

AI System Inventory

Model Intake Record

Model Provenance Record

RAG Source Inventory

Retrieval Authorization Test Record

Prompt Injection Test Record

Agent Tool Registry

Agent Blast-Radius Worksheet

Tool Permission Matrix

Human Approval Decision Record

Eval Gate Log

AI Vendor Intake Review

Governance Evidence Map

AI Incident Reconstruction Log

Synthetic Media Verification Record

Hardware Isolation Review

12. First-Hire 30/60/90-Day Plan

Example

First 30 Days

The first AI security hire should build visibility and credibility before attempting broad process change. Milestones: create an initial AI system inventory, meet product engineering leads, identify the top five AI-enabled systems or pilots, review existing AI policies, collect current customer AI security questions, and document immediate high-risk gaps. Deliverables by day 30: initial inventory, stakeholder map, top-risk system list, and proposed 60-day review plan.

Days 31-60

The second phase should produce first controls and evidence. Milestones: run threat models for the top two high-risk systems, define model intake requirements, draft RAG and agent review checklists, identify required eval coverage, and align with GRC on evidence storage. Deliverables by day 60: two threat models, draft control registry, initial eval or red-team plan, model intake checklist, and first executive risk summary.

Days 61-90

The third phase should turn early work into cadence. Milestones: establish AI intake, define release gate triggers, start monthly evidence review, create risk acceptance format, align with procurement on AI vendor addendum, and propose hiring or contractor needs. Deliverables by day 90: operating cadence calendar, release gate matrix, control registry v1, vendor AI checklist, quarterly operating review agenda, and staffing recommendation.

Adaptation note

For a first hire focused on red teaming, replace threat models with scoped red-team exercises and eval conversion. For a governance evidence hire, emphasize inventory, control registry, evidence taxonomy, and executive reporting. For an agent security hire, prioritize tool inventory, permission matrix, and audit trace requirements.

13. AI Security Operating Cadence

Example

Weekly

• AI intake triage for new features, model changes, retrieval changes, tool additions, and vendor AI requests.
• Release blocker review for high-risk launches.
• Remediation follow-up for critical and high AI security findings.
• Office hours for product and engineering teams.

Weekly outputs: updated intake queue, launch decisions, blocker list, owner assignments.

Monthly

• Evidence freshness review across inventory, evals, model intake, retrieval authorization, agent controls, and vendor AI reviews.
• Metrics review for eval pass/fail trends, release blocks, incident triage, open risk acceptances, and unowned controls.
• Control registry update with new systems, closed gaps, stale evidence, and new exceptions.
• AI vendor change review with procurement and legal.

Monthly outputs: evidence scorecard, metrics snapshot, control registry update, vendor risk changes.

Quarterly

• AI security operating review with CISO, product, engineering, GRC, privacy, procurement, and legal.
• Red-team and eval roadmap refresh.
• High-risk AI system review.
• Staffing, tooling, and budget review.
• Executive and board reporting update.

Quarterly outputs: operating review deck, risk acceptance review, roadmap update, maturity assessment, staffing recommendation.

Adaptation note

Keep the cadence small at first. A lightweight cadence that actually happens is better than a mature-looking process that collapses after one month. The test is whether decisions, evidence, and owners become clearer every cycle.