AI Security Engineering Handbook

• Explain what belongs in an AI system inventory.
• Describe risk tiering criteria for AI-enabled systems.
• Connect inventory records to release gates and evidence.

Every AI security decision depends on knowing what exists, who owns it, and what authority it has. If the inventory is stale, threat modeling, vendor review, release gating, and incident scope all start from fiction. The model may have changed, a provider may have been added, or a retrieval index may now sit outside the original review. Inventory is not paperwork; it is the prerequisite for every other AI security control.

System Mechanics

An AI system is a deployable unit — a feature, workflow, API integration, or product — that uses one or more models to perform a meaningful function. A single business product may contain several distinct AI systems: a support summarizer, an action recommender, and an automatic reply drafter are three separate systems even if they share a provider.

The key distinctions are:

• Feature vs. system: a feature is the user-visible capability. A system is the technical deployment, with its own model, retrieval sources, tools, data handling, and risk surface.
• System vs. model: one system may call multiple models. One model may power multiple systems. Track both.
• Deployment vs. provider: the provider hosts the model infrastructure. The deployment is the organization's configuration, including the endpoint URL, API key scope, retrieval index, prompt templates, and tool definitions.

Inventory must capture these distinctions because security controls apply at different levels. Vendor risk applies to the provider. Behavioral testing applies to the model version. Authorization review applies to the deployment. Data handling review applies to the data categories the system touches.

Systems also have lifecycle states: proposed, experimental, approved for production, restricted (incident or policy hold), deprecated, and retired. Controls and evidence requirements differ by state. A system in experimental state may have lighter gates; a restricted system may need immediate telemetry review before returning to production.

Change triggers — events that require an inventory update — include: model version change, provider change, new retrieval source, new tool connection, user population expansion, new deployment region, architecture change, and post-incident remediation. The inventory program must define these triggers explicitly or records go stale between reviews.

• Map model, app, retrieval, tool, identity, provider, and telemetry boundaries.
• Explain how AI trust boundaries differ from ordinary application diagrams.
• Identify which evidence belongs to each boundary.

The most expensive AI security mistakes are architectural because they show up after the design has shipped, and the fix now requires rework. A team that asks where the design places trust before building usually produces a safer system than one that patches controls onto a finished product.

System Mechanics

An AI system involves four distinct flows, each with separate security implications:

Data flow carries information from sources to destinations — from the user to the application, from the database to the retrieval service, from retrieved chunks into the model prompt, from the model's output to downstream consumers. Data flow is what most security practitioners think of first.

Instruction flow carries behavioral directives — the system prompt, developer instructions, tool definitions, and policy constraints that shape what the model is expected to do. These directives have intended authority over the model's behavior.

Control flow determines execution sequencing — which function runs, which tool is called, which branch executes. In traditional software, control flow is fully deterministic. In AI systems, the model's output can influence control flow (by proposing tool calls), which makes the boundary between data and control non-deterministic.

Authority flow tracks where the right to perform an action originates and how it is delegated. A user has authority over their own data. An application holds authority from the user via a session. A tool executes under a service identity. The key insight: authority comes from the application's authorization layer and the execution identity's credential scope — not from the content of the model's output.

A trust boundary exists wherever an enforcement check must occur because the principal, privilege level, or data classification changes. Examples in an AI system: the edge between user input and system instructions (a user cannot elevate their message to system-instruction authority), the edge between retrieval results and authorized content (semantic relevance does not grant access), the edge between model output and tool execution (the model's proposal does not self-authorize), and the edge between the application and an external provider (data handling obligations apply).

The product security surface reaches far beyond the model. Prompt and context assembly, retrieval pipelines, tool and API integrations, authorization and identity controls, and logging all sit in the product boundary. Each is a distinct attack surface requiring its own control model.

• Identify AI-specific assets, attackers, abuse paths, and trust changes.
• Translate threat model findings into controls and release decisions.
• Use careful evidence language for uncertain AI behavior.

AI threat modeling almost always starts late. By the time security enters the room, the team has a model provider, a prompt template, a vector index, and a working demo. Decisions about what data the model can see, what tools it can call, and whether retrieved content might carry hostile instructions feel already settled. The question is not whether to do the analysis — it is how to do it effectively even when the design has momentum and the launch date is fixed.

System Mechanics

A threat model converts a system architecture description into a structured analysis of what can go wrong, why it matters, where controls belong, and how to prove they work.

The process begins with a system walk-through: drawing the data flow from user input through every application component, retrieval service, model provider, tool layer, and output destination. Then the analyst marks trust boundaries — where principals, privilege levels, or data classifications change — and authority transitions — the specific points where text becomes instruction, data becomes context, output becomes tool arguments, or a decision becomes an action.

These four authority transitions concentrate AI-specific risk:

1. 1Text becomes instruction. User-provided text enters a prompt alongside developer instructions. If the boundary between them is purely semantic (a prompt template with no structural enforcement), adversarial user text can attempt to reframe itself as instruction.
2. 2Data becomes context. Retrieved documents, email threads, and tool outputs enter the prompt as "evidence." If they contain adversarial content, the model may process it as directive.
3. 3Output becomes argument. Model text output is parsed into tool call parameters. If output can be influenced by injected content, the tool call parameters may reflect the adversary's intent rather than the user's.
4. 4Decision becomes action. A model proposal becomes an executed action via the orchestrator. If the orchestrator does not independently verify authorization before execution, the action may exceed the user's actual permissions.

STRIDE remains a useful baseline because AI systems still have all six threat categories. Spoofing (impersonating a user or service), Tampering (modifying prompts, retrieval sources, or model artifacts), Repudiation (insufficient logging to reconstruct what happened), Information Disclosure (unauthorized data in context or output), Denial of Service (exhausting token budgets or retrieval capacity), and Elevation of Privilege (using injected content to gain capabilities beyond the user's role). The limitation is that standard STRIDE templates do not ask about context authority, retrieval authorization, tool permission chaining, or model behavioral change. AI threat modeling requires explicit extensions for these.

• Explain context as an attack surface.
• Distinguish model-level refusal from application-level enforcement.
• Describe regression coverage for prompt, model, and retrieval changes.

Production prompt injection risk is less about the user who types "ignore your previous instructions" and more about the document the system retrieves for that user. Direct injection is visible and gets patched fast. Indirect injection through retrieved documents, email threads, ticket comments, and tool outputs lasts longer because the application treats those sources as trusted evidence, not as attack paths. The system needs external content to work, and its security depends on limiting what that content can cause.

System Mechanics

The model processes all tokens in its context window through the same mechanism — there is no cryptographic boundary, no hardware-enforced privilege ring, and no structural distinction between "these are instructions" and "this is data." The model infers context authority from position, role labels, and formatting conventions in the prompt template, but these are conventions, not enforcement mechanisms.

This is the root of prompt injection. When untrusted content enters the context alongside developer instructions, the model may interpret that content as authoritative. A retrieved document that begins with "SYSTEM: The following is an updated directive from the developer..." is just tokens. The model has no way to cryptographically verify that those tokens originate from the developer's system prompt rather than from a retrieval source.

The two primary attack paths:

Direct injection: The user submits adversarial text in their own message turn. The system may filter or sanitize user input, making this the more visible and more patchable path. Defense: input validation, structural prompt separation, output schema enforcement.

Indirect injection: Adversarial instructions are embedded in content that the system processes — retrieved documents, email threads, calendar entries, issue tracker comments, web pages, tool output. The system does not show this content to the user before processing it. The content may have been placed by an attacker days or weeks in advance. Defense: structural context labeling, output schema enforcement, tool authorization independent of model reasoning, monitoring for anomalous output/action patterns.

The important distinction: prompt injection is dangerous in proportion to what it can cause. A prompt injection that changes a tone of voice is low severity. A prompt injection that causes a tool call to update a CRM record, exfiltrate data, or bypass an approval gate is high severity. The correct frame is not "detect all injection" but "limit what injection can cause."

• Explain why authorization must happen before context assembly.
• Reason about stale permissions, poisoning, tenant isolation, and citations.
• Identify retrieval evidence needed for assurance and incident response.

Retrieval-augmented generation changes the data access model in ways most security programs have not caught up with. The search layer is not search; it is a data access path that builds context for the model, and it needs the same rigor as any other sensitive-data path. The failure teams discover in production is simple: they built access for the answer by checking what the model may say while leaving search mostly open.

The authorization boundary must be enforced before results enter the model's context. A filter that runs after similarity ranking has already broken that boundary. The model processes whatever the index returns, regardless of what the output shows. The filter gate is not an optimization — it is the control.

System Mechanics

The RAG lifecycle has two phases: ingestion (building the index) and retrieval (serving queries). Security failures can originate in either phase.

Ingestion phase:

1. 1Source collection — documents are gathered from source systems (wikis, CRMs, file systems, email, tickets). Each source has its own permission model.
2. 2Parsing — documents are converted from their native format (PDF, HTML, Markdown) to plain text. Formatting metadata may be lost here.
3. 3Chunking — documents are split into smaller segments (chunks) suitable for embedding. Permission and ownership metadata must survive chunking.
4. 4Embedding — each chunk is converted into a numerical vector by an embedding model. The vector captures semantic meaning but not access control.
5. 5Indexing — vectors and associated metadata (source ID, owner, tenant, classification, access policy, removal status) are stored in the vector index. The metadata stored here is what retrieval-time authorization queries.

Retrieval phase:

1. 1Query — the user's request (or a reformulation of it) is embedded into a query vector.
2. 2Eligibility filtering — before similarity search, the query is accompanied by mandatory filters: tenant ID, user role, document classification floor, purpose. Only chunks matching all filters are candidates. This is the primary authorization enforcement point.
3. 3Retrieval — among eligible chunks, the index finds those most semantically similar to the query vector.
4. 4Reranking — a secondary scoring model or function re-orders the retrieved chunks by quality. Reranking does not re-evaluate authorization.
5. 5Context construction — top-ranked eligible chunks are included in the model prompt as retrieved context.
6. 6Generation — the model generates a response grounded in the retrieved content.

The critical insight: steps 7 and 8 must occur in this order. Filtering must precede similarity search. An implementation that retrieves by similarity first and then filters by eligibility has allowed unauthorized content into the ranking computation — a subtler boundary violation that can still produce information leaks via reranking score patterns.

• Classify tool permissions and side effects.
• Explain why approvals require context and runtime enforcement.
• Reason about action chains, identity, auditability, and rollback.

The security model for agents breaks down fast when one confused or compromised model call can write to email, source code, cloud resources, issue trackers, calendars, or customer records. For a text assistant, the failure may stay inside the interface. For an agent, one injected instruction in a retrieved document can become a company-wide incident. That gap is the scope of agent security.

System Mechanics

An agent operates in a loop. Understanding the loop is prerequisite to designing controls around it:

1. 1Request — the user submits a goal or task. This establishes the authorized scope: what the user asked for.
2. 2Model proposal — the model processes the current context (system prompt, user request, tool definitions, conversation history, prior tool results) and generates a response. If the task requires an action, the response contains a structured tool call proposal — a JSON-formatted signal naming a tool and its arguments.
3. 3Structured tool call — the model's output is data, not a command. The orchestrator reads the proposed tool call.
4. 4Orchestration — the orchestrator evaluates the proposal. Does the tool exist? Is this tool permitted in the current context? Do the arguments fall within allowed scope? Is approval required?
5. 5Policy decision — if the orchestrator's policy checks pass, execution proceeds. If they fail, the model is informed and may propose an alternative or terminate.
6. 6Execution identity — when approved, the orchestrator invokes the tool using a scoped service credential. This credential — not the model's output — defines what the tool can actually do. The credential's scope is the blast radius floor.
7. 7Tool action — the tool executes: reads data, writes a record, sends a message, runs a command, calls an API.
8. 8Returned result — the tool's output is passed back to the model as new context. Tool output is untrusted content — the same caution applies as to retrieved documents.
9. 9Subsequent calls — the model may propose additional tool calls. Each must pass through the same policy gate. Actions accumulate; blast radius grows with each step.
10. 10Final output or side effect — the loop terminates when the model produces a final response, when a termination condition fires, or when a policy gate stops it.

The key security insight: the model proposes; the orchestrator decides. Authority comes from the application's credential configuration and policy checks — not from what the model's output says. A well-formed tool call proposal from a model that was misled by injected content does not become authorized merely because it is well-formed.

Every agent interaction follows a delegated action chain. A user prompt becomes model reasoning. Model reasoning produces tool arguments. Tool execution changes real-world state. The security review must trace the full path from prompt to side effect, not stop at the model response.

The difference between an AI assistant and an AI agent is blast radius. An assistant's worst outcome is a bad answer inside the user interface. An agent with write access to email, cloud infrastructure, and production data can cause company-wide damage from one misled model call.

• Identify sensitive data paths in AI workflows.
• Explain minimization, retention, logging, and deletion evidence.
• Connect privacy obligations to engineering controls.

The privacy problem in AI systems is derived data. A customer support message can become a fine-tuning example, then an embedding, then an eval fixture, then an inference-time search result. Each step creates a new record with its own retention, access, and removal rules. Traditional privacy programs were built around database rows. AI systems add vector indexes, model weights, prompt logs, and annotation queues that those programs were never designed to govern.

System Mechanics

Personal and sensitive data in AI systems does not follow a single path. It moves through a lifecycle with AI-specific transformations that create derived artifacts, each with its own retention, access, and removal requirements.

The data lifecycle for an AI feature typically covers:

1. 1Source — the original data (customer message, document, user record) in its native system.
2. 2Transit — data in motion to the application, over API calls, to the provider's inference endpoint. Encryption in transit is baseline; note that data leaves the organization's network at the provider boundary.
3. 3Prompt and context — data assembled into a model prompt. This is often a transient representation, but if logged it becomes a persistent record containing everything the model saw.
4. 4Context window at provider — during inference, the data is processed by the provider's infrastructure. Provider data handling terms govern what the provider retains, for how long, and for what purpose.
5. 5Cache — responses or context may be cached for performance. Cached data may persist longer than the session and may contain sensitive content.
6. 6Log — prompt and response logs are the primary forensic artifact for AI systems. They may contain personal data, secrets pasted into context, health information, or financial data. Logs are a new category of sensitive data store.
7. 7Embedding — when data is embedded for RAG indexing, it is transformed into a numerical vector. Embeddings are not human-readable but may still support re-identification and may retain sensitive content in a form that is difficult to audit.
8. 8Dataset — data assembled for fine-tuning or evaluation. Each dataset has its own legal basis requirement and must honor removal requests.
9. 9Derivative — model weights incorporate training data patterns. Memorization is a documented risk: models can reproduce verbatim training content during inference. Cleaning trained-in data from model weights is generally not technically possible.
10. 10Deletion — removal must propagate across all derived forms. Deleting the source record does not remove the embedding, the prompt log, the cache entry, or the fine-tuning example.

• Separate model behavior risk from provider security risk.
• Identify vendor evidence needed for hosted AI dependencies.
• Explain why model updates require monitoring and change review.

Companies using external model APIs are in a dependency relationship most security programs have not fully mapped. The vendor controls model behavior, training data, safety settings, update cadence, routing, and data retention terms. The company controls the app layer and the context it sends. That boundary is where major AI security risk lives, and it gets less review than most third-party dependencies, partly because the API feels like infrastructure and partly because few teams have written the checklist.

System Mechanics

Model and provider risk operates across four separate layers. Conflating them leads to incomplete controls:

Layer 1 — Model capability and behavior risk: the model's responses, safety thresholds, and edge-case behavior can change without any change to the application code. A provider-side update may alter how the model handles adversarial prompts, whether it follows structured output constraints, or how it responds to injection attempts. The company cannot inspect the model weights or the training data. It can only observe behavioral outputs.

Layer 2 — Model artifact and provenance risk: for self-hosted models, the artifact itself (weights, adapters, tokenizer) is a supply-chain item. It can be tampered with, incorrectly sourced, or contain embedded risks in the serialization format. This layer applies to open-weight models and fine-tuned adapters — not to managed APIs where the provider holds the artifact.

Layer 3 — Provider security and operational risk: the provider holds the infrastructure, the model artifact, and the prompt/response data during inference. A provider security incident can expose the company's prompts, customers' data, and API credentials. Provider availability failures affect the company's product. The provider may route traffic through sub-processors that the company has not reviewed.

Layer 4 — Contractual and governance risk: data retention terms, training-on-input settings, sub-processor lists, audit rights, and incident notification timelines are contractual obligations. If the contract does not address them, the default terms apply — which may not match the company's privacy obligations or customer commitments.

These layers correspond to different organizational functions: engineering owns Layer 1 monitoring; security and ML platform own Layer 2 controls; vendor management and security own Layer 3 assessment; legal and procurement own Layer 4 review.

The transparency problem: managed API providers may not announce model version changes in ways that allow precise behavioral tracking. The application may be sending prompts to a model that has changed since the last evaluation — without any notification. This is not a policy failure by the provider; it is a structural characteristic of managed inference services. The response is behavioral monitoring, not reliance on change notifications.

• Trace model artifacts from source to production use.
• Identify intake, integrity, license, registry, and rollback evidence.
• Reason about unsafe formats, public hubs, and adapter risk.

The company that would never deploy a software dependency without reviewing its source, checking its hash, and verifying its license often deploys model weights from public hubs without those checks. The oversight is usually a category error, not negligence. The team that owns model deployment thinks in terms of performance and inference cost, not supply-chain trust. AI supply-chain security closes that gap before an incident makes it visible.

System Mechanics

AI supply chain security covers three categories of artifacts with different threat profiles:

Category 1 — Software and infrastructure: application code, package dependencies, containers, and the orchestration and CI/CD infrastructure running the AI system. These follow standard software supply chain controls: dependency scanning, SBOM, signed containers, pinned dependencies, and verified build pipelines. Standard software supply chain practices apply here without major AI-specific extension.

Category 2 — Model and ML artifacts: the artifacts that most distinguish AI supply chains from conventional software supply chains. This category includes:

• Model weights (the primary artifact — can be hundreds of gigabytes)
• Adapters (LoRA, QLoRA, prefix tuning weights — smaller, can be applied to a base model)
• Tokenizers (code that converts text to token IDs — can execute code if malicious)
• Loaders (Python code required to load some model formats — can execute arbitrary code)
• Embedding models (produce vectors from text — same provenance concerns as generative models)
• Datasets (training and evaluation data — can introduce poisoned behavior)
• Eval sets (the test suites that validate behavior — can create false confidence if tampered)

Category 3 — Operational configuration: prompts, tool definitions, retrieval source configurations, orchestration configuration, and vendor connection settings. These are often treated as application code, but they can be managed and versioned separately and can be tampered with or substituted outside normal code review.

The specific supply chain threat that has no direct software analogy: unsafe serialization formats. Some model artifact formats execute code during the loading process. The most common example is Python's pickle format, used by PyTorch checkpoints. Deserializing a pickle file can execute arbitrary code in the loading process's context — which in a model-serving environment typically has access to GPU resources, object stores, internal network, and production credentials. A malicious model artifact served from a public hub can compromise the inference server simply by being loaded. Safer formats such as safetensors eliminate this risk for weight tensors, but format safety is one control, not the complete supply chain program.

A model artifact earns live eligibility by moving through a governed lifecycle. Each stage produces evidence: intake review, hash verification, license review, registry entry, promotion gate, and deployment. That chain makes the supply chain auditable, reproducible, and defensible when a security question arises.

• Name the telemetry required for AI detection, forensics, and evidence.
• Explain log minimization and sensitive-data handling tradeoffs.
• Connect telemetry fields to investigations and control evidence.

The question most AI incident investigations cannot answer is: what was the model given? Standard app logs capture what the user sent and what the system returned. AI investigations also need what the system assembled and sent to the model: retrieved documents, conversation history, system instructions, and tool outputs. The gap between the network layer and the context window is where most investigations stall.

System Mechanics

AI telemetry requires understanding how observability concepts map to AI-specific events:

A log is a discrete, time-stamped record of an event. A metric is an aggregated measurement over time. A trace is an end-to-end record linking all events across components for a single request. A span is one operation within a trace (e.g., "retrieval query," "model call," "tool execution"). An evidence record is a log event retained specifically because it proves a control operated. An audit record is a tamper-evident record intended to survive governance, legal, or regulatory review.

For AI systems, these overlap but must be distinguished in the logging architecture. A model API call is both a span in a trace and potentially an audit record if it involves a high-risk action. A retrieval event is a span and an evidence record proving retrieval authorization occurred.

The forensic gap in most AI logging: application logs capture what the user sent and what the application returned. They do not capture what the application assembled and sent to the model — the full assembled prompt including system instructions, retrieved chunks, conversation history, and tool outputs. Without this context assembly record, incident investigation cannot answer "what did the model see?" — which is the first question in every AI security investigation.

A second gap: AI systems have multiple independent log sources — the application layer, the retrieval service, the model API, the tool execution layer, and the output filter. Without a shared correlation identifier (trace ID or session ID) flowing through all layers, reconstructing a single request across these sources requires manual reconciliation. In incident investigations, this reconciliation can take days.

Logging is not a free control. Prompt logs are a new category of sensitive data. A prompt log from a customer support assistant may contain: customer PII, health information, financial data, credentials pasted into context, and business-sensitive conversation content. Log access must be controlled, retention must be defined, and redaction must be specified — before the first log record is written.

• Map AI failure modes to observable signals.
• Explain coverage, alert quality, and false-positive tradeoffs.
• Connect detection findings to incident response and regression testing.

Anomaly detection without a baseline is pattern matching against noise. Many AI security programs invest in authorization, approval gates, logging, and release gates, then treat detection as something that happens during incident response rather than before it. That order guarantees that incidents are found by their effects. Detection engineering is the work of deciding, before incidents occur, what behavior points to a control failure, what logs capture it, and what response logic fires.

System Mechanics

Detection engineering follows a repeatable development lifecycle:

1. 1Hypothesis — name a specific control failure: "The retrieval authorization filter fails and returns a chunk from a different tenant."
2. 2Observable behavior — describe what the failure looks like in system behavior, not in output text: "Chunk IDs in the retrieval trace carry tenant metadata that does not match the requesting user's tenant."
3. 3Telemetry source — confirm the trace field exists: retrieval_span.chunk_tenant_id vs. session.tenant_id.
4. 4Rule logic — write the detection condition: alert when any chunk in a retrieval trace has chunk_tenant_id != session.tenant_id.
5. 5Threshold or sequence — some detections fire on a single event; others require a count or sequence (e.g., 3 out-of-scope tool calls within one session).
6. 6Enrichment — add context to the alert: user ID, session ID, tenant, chunk IDs, and the trace record that triggered it.
7. 7Triage — define the first three questions an analyst should ask when the alert fires.
8. 8Validation — test the rule against historical data (should fire on known incidents) and against synthetic normal traffic (should not fire on normal behavior).
9. 9Tuning — measure false-positive rate during calibration; adjust thresholds.
10. 10Response mapping — define the playbook triggered by this alert.
11. 11Feedback — after any incident related to this control, review whether the rule fired at the right time and whether it needs to change.

The key principle: AI detection targets control failures, not adversarial content. Trying to detect the text of an injection attempt is difficult and brittle — the attack space is unbounded. Detecting the consequences of a control failure is more reliable: unauthorized chunk in retrieval results, tool call with arguments sourced from retrieved content rather than user input, output schema deviation, approval gate bypass, unusual tool call sequence. These are structural signals, not content signals.

• Classify AI incidents by failure class and affected boundary.
• Explain containment options for retrieval, agents, providers, and prompts.
• Describe the evidence needed to reconstruct an AI incident.

Containment decisions made without context are guesses with consequences. AI incident response differs from standard incident response in one key way: scope and severity depend on live context state, not only on code version or deploy history. A prompt injection incident may affect only the sessions that retrieved one poisoned document in one time window. A retrieval authorization failure may affect only users in one tenant while the index was in one state. A model update drift may affect only requests that matched one behavior after a provider routing change. Scope needs context-aware logs, not a count of records changed since the last deploy.

System Mechanics

AI incident response differs from standard incident response in one critical dimension: scope and severity depend on the context state at the time of the incident, not just on what version of code was deployed. Understanding this requires understanding the AI-specific incident lifecycle:

Preparation — before incidents, build playbooks for each failure class, confirm all AI-specific containment actions are operational (not just documented), verify access controls allow responders to execute containment without multi-hour approval chains, and run tabletop exercises.

Identification — detection alerts or external reports surface the incident. The first triage question is not "was this a security incident?" but "what does the context chain show?"

Triage — before classification, rebuild the context chain: what did the user request, what was assembled in the prompt, what was retrieved and authorized, what did the model receive, what tools were proposed and executed, and what was the output and any side effect. The label — prompt injection, retrieval authorization failure, unauthorized agent action, model drift, supply chain — follows from the chain. Output content alone is insufficient for triage.

Containment — AI-specific containment options go beyond standard code/credential revocation. They include: removing a poisoned document from the retrieval index, suspending a prompt template version, disabling a specific tool connector, revoking an agent OAuth token, rolling back to a pinned model version, invalidating cached responses from a time window, forcing human approval for subsequent sessions, or disabling the feature.

Evidence preservation — before remediation, preserve: the assembled context (prompt content if logged), retrieved document IDs and versions, tool call records and arguments, authorization decision records, approval records, model version metadata, and output records. These are the artifacts that prove what happened and what the model was given.

Eradication and recovery — address the persistence mechanism, not just the immediate session. A retrieval injection is not contained until the poisoned document is removed from the corpus and removal is confirmed.

Lessons learned — every incident should produce: a new or updated detection rule with test cases, a trace field addition or correction, an architecture change with threat model justification, or a playbook update. Each improvement is tracked to completion.

• Describe the difference between demos, evals, red teaming, and regression tests.
• Explain how findings become closure and release evidence.
• Use severity and coverage language without overclaiming.

Most AI red-team exercises produce a report. The report lists what the team found, maybe includes screenshots, and recommends fixes. Then the assessed team decides what matters. That is not adversarial evaluation. It is advice with a dramatic look. The difference between a red-team exercise and an adversarial control is whether findings become regression tests, whether those tests block future releases, and whether closure needs evidence rather than conversation.

System Mechanics

Evaluation and regression testing for AI systems differs fundamentally from conventional software testing because AI behavior is probabilistic, not deterministic.

Deterministic tests assert a fixed outcome: given this exact input and context, the output must match this exact criterion (e.g., "the output must not contain any chunk_id belonging to a different tenant"). These are the most reliable tests and should cover control-enforced behaviors — authorization decisions, schema validation, tool call blocking.

Probabilistic tests measure behavior across multiple samples: given this input and context, run the model N times and require that X% of responses meet the criterion. This is necessary for evaluating behaviors that the model handles probabilistically — adversarial prompt handling, refusal rates, output quality. A single passing response proves nothing about population behavior at scale.

Repeated sampling is the mechanism: run the same test case 10, 20, or 50 times. Count the failure rate. Compare against a defined threshold (e.g., "must not produce a policy-violating response more than 5% of the time"). The threshold is a security design decision, not a default.

Evaluator models (LLM-as-judge) use a second model to assess whether the first model's output meets a criterion. This is useful for open-ended quality questions but has important limitations:

• Evaluator bias: the judge model has its own tendencies that may not match the criterion.
• Correlated failure: if the judge and the tested model share architecture or training data, they may fail together.
• Calibration drift: the judge's judgments may shift as the judge model is updated.
• Prompt sensitivity: different judge prompts for the "same" criterion can produce different verdicts.
• Ground truth gap: the judge's verdict is not ground truth — it is an automated opinion. High-impact cases require human review.

Release gates are the mechanism that gives evaluations operational force. An eval becomes a control when: failing it blocks the release (not just generates a report), the gate is enforced in CI/CD or deployment tooling, the failure consequence is defined in advance, and exception requires explicit risk acceptance documentation.

An eval program is not a one-time exercise. It is a continuous control loop: run tests, identify findings, convert findings into regression cases, update release gates, and repeat. The loop's value compounds with each iteration as the test suite grows to cover discovered failure classes and the release gate reflects current risk knowledge.

• Translate governance expectations into engineering artifacts.
• Explain evidence freshness, owner accountability, and claim-readiness.
• Separate policy language from controls that operate.

Governance is only real when it can answer three questions without hesitation: which systems are in production, who owns each control, and what evidence proves the control worked. If it cannot answer those questions, the program has a policy problem, not a documentation problem. Frameworks like NIST AI RMF, ISO 42001, and OWASP LLM Top 10 describe mature oversight. They do not generate the artifacts. That work is engineering.

System Mechanics

Governance programs for AI systems fail at a predictable structural point: the translation step between framework obligation and engineering artifact.

A framework like NIST AI RMF or ISO 42001 describes what mature AI governance looks like. It does not generate the artifacts that prove it. The translation chain has four steps, and each step can break:

Step 1: Obligation identification — which framework requirement applies, and how does it apply to this system specifically? A generic "monitor AI systems for harmful outputs" requirement means different things for a customer-facing chat assistant and an internal code generation tool. If the obligation is not made system-specific, it cannot be owned.

Step 2: Control objective definition — what system behavior, engineering practice, or operating procedure would satisfy the obligation? Control objectives must be testable: not "we monitor AI systems" but "the Nexus Support Assistant has an automated eval suite running 40 security-relevant cases weekly against the live endpoint, with a defined alerting threshold and named on-call owner."

Step 3: Control ownership — who operates the control, produces the evidence, and responds when the control fails? Committee ownership fails because committees cannot run eval suites, review trace logs, or update detection rules. Control ownership requires a named team with operational capability.

Step 4: Evidence production and retention — what artifact proves the control operated? Evidence requirements differ by control type. For an eval gate: the eval run record showing model version, test cases, pass/fail result, and release decision. For a vendor review: the completed intake checklist with all required fields, signed by the named reviewer. For an incident response exercise: the tabletop exercise record showing scenario, participants, gaps identified, and remediation tasks.

The chain from framework obligation to audit record only holds if all four steps are completed for every control. A gap at any step means the control is either unowned, unoperational, or unevidenced.

Customer security questionnaires represent the governance-to-customer trust direction. The risk for AI features: teams overclaim maturity they cannot prove, or underclaim capabilities they have built. The discipline is to answer each question with the evidence artifact that proves the claim, not with aspirational policy language.

Governance works in both directions. Policy intent must move down through control owners to engineering tests and technical evidence. Evidence must move back up to satisfy audit duties and inform executive decisions. The chain from policy to audit record is only as strong as the translation steps in between.

A mature AI security function runs on three interlocking cadences: weekly intake and triage keep current deployments governed and new deployments from slipping through intake, monthly evidence and gap review track control freshness and surface failures before incidents make them visible, and quarterly strategy and reporting connect the operating model to leadership decisions and external obligations.

This crosswalk is an engineering evidence map, not legal advice. It uses broad framework themes and maps them to artifacts that help a security team prove control operation. Legal, compliance, and privacy teams should validate jurisdiction-specific obligations before public claims are made.

Synthetic media risk belongs in the handbook because it creates security decisions, not communications risk. Deepfake voice calls, synthetic interview candidates, manipulated customer media, forged approval evidence, and generated documents can all enter security workflows. The control question is not whether a team can perfectly detect synthetic content. The control question is whether high-impact decisions rely on media or identity evidence without an independent check path.

Start by identifying workflows where audio, video, images, or remote identity signals can authorize action or influence trust: executive approvals, payment changes, hiring interviews, customer onboarding, account recovery, fraud review, incident escalation, vendor instructions, and legal or compliance evidence. For each workflow, define which media is advisory, which media is evidence, and which media can trigger action.

Minimum viable controls include out-of-band checks for high-risk approvals, liveness checks for identity proofing, known-channel callback procedures, dual approval for unusual financial or access requests, origin or watermark review where available, vendor claims review, and incident handling for suspected synthetic media.

Evidence artifacts should be lightweight but explicit. A Synthetic Media Verification Record should capture the asset type, workflow, check method, reviewer, decision, and evidence retained. A Watermark Verification Log can record whether watermark, origin, or content-authenticity signals were checked and what they proved. A Liveness and Identity Verification Review should capture the identity workflow, vendor control, fallback process, false-accept concern, and escalation path.

Do not overclaim detection certainty, use careful language: the company applies check controls, reviews origin signals where available, requires out-of-band confirmation for high-risk actions, and records evidence for investigation.