Designing Secure AI Architectures

AI systems inherit every security concern of conventional software and add new ones. They process untrusted natural language as if it were instructions, they often aggregate data from many sources into a single index, and they increasingly take actions through tools. Securing them is less about new products and more about applying established principles, least privilege, defense in depth and explicit trust boundaries, to a new kind of component.

01Threat model the system, not just the model

Security reviews of AI systems sometimes focus narrowly on the model: can it be jailbroken, will it produce harmful output. Those questions matter, but much of the material risk sits in the surrounding architecture. A useful threat model traces data and control flow through every component:

• Where does data enter: user prompts, uploaded files, retrieved documents, tool results, web content?
• What can the model reach: which indexes, which tools, with whose credentials?
• Where does output go: back to the user, into another system, into an email, into code that will run?
• What is logged, where, for how long and who can read it?

Public frameworks are useful prompts for this work. The OWASP Top 10 for LLM Applications and MITRE ATLAS both catalogue attack techniques specific to AI systems, and they fit naturally into a conventional threat modeling exercise such as STRIDE.

02Data boundaries in retrieval

Retrieval-augmented generation concentrates data. Documents that were spread across systems, each with its own access controls, end up in one index. If the index does not carry those controls, the assistant becomes a way to read anything in it.

• Capture source permissions at ingestion and store them with each chunk.
• Filter by the requesting user's identity inside the retrieval query, before results reach the model. Filtering generated answers afterwards is too late.
• Re-check access at query time for sensitive sources, or fail closed when permission sync is stale.
• Classify sensitive data before indexing, and exclude or redact it where required, instead of relying on the model to withhold it.
• Separate indexes by tenant or sensitivity tier when filtering alone is not a strong enough boundary.

03Identity for models and agents

Every call an AI system makes to another system should be attributable to an identity with the minimum necessary permissions. Two patterns matter.

Delegated user identity

When an assistant acts for a user, it should use credentials derived from that user's session, such as an OAuth token with narrowed scopes, so it cannot exceed the user's own access. This also keeps downstream audit logs meaningful.

Workload identity for the system

Background agents and pipelines need identities of their own, issued by the platform rather than stored as long-lived keys, scoped to specific resources and rotated automatically. Avoid shared service accounts that accumulate permissions over time.

04Prompt injection is an architecture problem

Prompt injection occurs when content the model processes contains instructions that override the developer's intent. Direct injection comes from the user; indirect injection arrives inside documents, emails, web pages or tool results. There is currently no reliable way to make a model ignore every injected instruction, so controls should assume some will succeed and limit what they can achieve.

• Constrain capabilities after exposure to untrusted content. Once an agent has read external input, remove or gate tools that send data out or modify records for the rest of that task.
• Require human confirmation for consequential actions, showing the exact action and its target.
• Allow-list outbound destinations, and prevent the model from constructing arbitrary URLs.
• Treat model output as untrusted input to every downstream system: escape it before rendering and never pass it unvalidated to a shell, a query or an interpreter.
• Use injection classifiers as an additional layer and log their verdicts, but do not make them the only control.