BlogCybersecurity and AI governance

LLM application security: a practical checklist

An LLM feature is an ordinary application with one unusual part: a model that will, sooner or later, be talked into doing the wrong thing. Securing it means building everything around the model so that nothing important breaks when that happens. This checklist follows a feature from design to operation and is based on OWASP’s 2026 lists for LLM and agentic applications.

What do OWASP’s current lists say?

The OWASP GenAI Security Project reissued its Top 10 for LLM Applications as a 2026 edition in August. For the first time the ranking was tested against a record of real incidents as well as a practitioner vote, and the order moved. Prompt injection stays first and sensitive information disclosure second. Excessive agency, meaning a model with more tools, permissions or autonomy than its job needs, rose to third. Unbounded consumption climbed to sixth and now puts cost at its centre. System prompt leakage was widened into hidden context exposure. Improper output handling fell to tenth, which we would not read as permission to relax: it is still how a manipulated answer becomes a compromised browser or database.

The project leads open their letter with a line that sums up the document: ‘Stop trying to build a model that cannot be fooled.’ Build the system around it instead.

The list also draws a boundary. It covers the model as a component inside your application. Once the model calls tools, carries memory between sessions and sets things in motion, the Top 10 for Agentic Applications, published in December 2025, applies too. Most features we build now sit on that boundary, so we work from both.

OWASP LLM entry (2026) The control that does most of the work
LLM01 Prompt Injection Limit what a fooled model can reach and do
LLM02 Sensitive Information Disclosure Send less, authorise before retrieval, scrub logs
LLM03 Excessive Agency Fewer, narrower tools that run with the user’s permissions
LLM04 Supply Chain Pin and review models, packages, tool servers and prompts
LLM05 Data and Model Poisoning Control who can write to training data, indexes and memory
LLM06 Unbounded Consumption Token quotas and hard spending caps
LLM07 Misinformation Ground answers and verify them before acting
LLM08 Hidden Context Exposure No secrets or access rules in prompts
LLM09 Vector and Embedding Weaknesses Tenant and permission filters inside the index query
LLM10 Improper Output Handling Treat model output as untrusted wherever it lands

A top ten is a prompt for thinking and a poor compliance target. Real incidents rarely fit one row. They chain three.

Design: threat model the feature

Take a commercial insurance broker adding an assistant to its customer portal. It answers questions from policy documents, looks up the signed-in customer’s policies and claims, reads documents the customer uploads, and can send a message to the claims team.

A threat model for this takes an hour and four questions.

What text reaches the model, and who wrote it? The system prompt, the customer’s message, policy wordings, uploaded claim documents, tool results. Only the first is written by you. An uploaded PDF was written by whoever produced it, and that may not be the customer.

What can the model reach? Which data, through which tools, with whose permissions.

Where does its output go? A browser, an email to staff, a database field, another system.

What does a failure cost? A wrong answer about cover, another customer’s claim on screen, a message sent that should not have been.

The design checklist:

  1. List every source of text in the context and mark it trusted or untrusted. Trusted means the system prompt, developer configuration and the authenticated user’s own request. Everything else is untrusted, including retrieved documents and tool results.
  2. List every tool with the worst outcome of a call made with hostile arguments.
  3. Look for sessions that combine private data, untrusted content and a way to send data out. OWASP’s 2026 text treats that combination as a check to make before deployment. Where all three meet, split the session or put a person in front of the outward step.
  4. Decide what the model will never be trusted with: access decisions, secrets, and anything you would mind a user reading. Assume the whole context window can be discovered.
  5. Classify the data that will enter prompts and decide what may go to the model provider. We cover that in data privacy with LLMs.
  6. Write abuse cases next to the user stories, and name an owner for the feature’s security.

For the broker, step 3 finds the problem early. Uploaded documents (untrusted), claims data (private) and the message tool (outward) share a session. The fix is cheap at design time: messages go to the claims team’s fixed queue and never to an address the model supplies.

Build: inputs, outputs and tools

Inputs and prompt injection

No filter reliably stops prompt injection, and OWASP’s current text says so plainly: the defence has to be architectural. Do the inexpensive things anyway. Keep untrusted content in a clearly marked part of the prompt. Strip invisible Unicode characters on the way in and on the way out. Filter what you extract from images and audio as well as typed text. Cap input size. Then assume all of it fails some day and rely on the rest of this list. The layered defence is the subject of guardrails for AI agents.

Outputs: treat the model as an untrusted user

Whatever the model writes may have been steered by someone else’s text. Its output gets the same handling as input from the internet, at every place it lands.

  • HTML and Markdown. Encode output for the context it is displayed in and set a strict content security policy. Stop chat clients loading images or link previews from model output automatically, because an image address can carry data out in its query string.
  • SQL. Parameterised queries only. If the model must write queries, give it a read-only role over restricted views, with a statement timeout. A fixed set of query templates that the model chooses between is safer still.
  • Shell and code. Avoid it. Where the product needs it, run the code in a sandbox with no credentials and a network allow-list.
  • Tool arguments. Validate against a schema in application code and check authorisation again at the moment of execution.
  • Terminals and logs. Strip control characters before writing model output to anything that interprets them.

Tools: who is calling, and may they?

The tool decides, not the prompt. Identity comes from the session, never from an argument the model fills in:

def get_claim(args, session):
    claim_id = ClaimId.parse(args["claim_id"])      # schema check, in code
    claim = claims.fetch(claim_id)
    if claim.customer_id != session.customer_id:    # authorisation, in code
        raise Forbidden()
    return claim.summary()                          # only the fields an answer needs

A line in the system prompt saying ‘only discuss the current customer’s claims’ is a request. The if statement is a control. OWASP calls this complete mediation, and its AI agent security cheat sheet makes the same point about enforcing authorisation outside the agent’s context.

Build checklist, part one:

  1. Model output is encoded, parameterised or sandboxed at every sink.
  2. Every tool has a strict input schema and validates it in code.
  3. Every tool call runs with the signed-in user’s identity and is authorised in the tool or the system behind it.
  4. There are no open-ended tools: no ‘run SQL’, ‘fetch any URL’ or ‘email anyone’.
  5. Irreversible or external actions wait for a person, who is shown the exact action.

How an agent should hold credentials at all is covered in identity and access for AI agents.

Build: secrets, data and supply chain

Secrets and keys

Nothing secret goes in a prompt. Connection strings, API keys and internal rules about who may see what will be extracted eventually. Model API keys live in a secrets manager, one per environment and service, each with its own spending limit, and are never shipped in a browser or mobile client. The model never handles a credential. Tools hold them.

Data leaking through prompts and logs

Send the fields the task needs, not the whole record. In retrieval, apply the user’s permissions inside the index query, before anything reaches the prompt. A filter applied afterwards has already put the data in front of the model.

Then look at your logs. Traces contain full prompts, retrieved passages and tool results, which makes the observability stack a second copy of your most sensitive data, often with wider access than the database it came from. Redact before export, set a retention period and restrict who can read them.

Supply chain

An LLM application depends on more than packages.

  • Models. Use sources you can verify, pin exact version identifiers and rerun your evaluations when a version changes.
  • Packages. Scan as usual, and check that a dependency suggested by a coding assistant exists and is the one you meant.
  • Tool servers. An MCP server is third-party code and third-party text. Pin it and review its tool descriptions, as set out in MCP security.
  • Prompts from outside. A prompt template, agent configuration or ‘skill’ copied from the internet is code. Review it as code.

Build checklist, part two:

  1. No secrets or access rules in prompts; model keys sit in a secrets manager with a limit on each.
  2. Retrieval filters by user and tenant inside the query.
  3. Prompts and traces are redacted, kept for a fixed period and access-controlled.
  4. Models, packages, tool servers and imported prompts are inventoried, pinned and reviewed.

Launch: limits, tests and a kill switch

Model calls cost money per token, so attacks on availability have become attacks on the bill, which OWASP’s text calls denial of wallet. Its unbounded consumption entry asks for limits counted in tokens and cost as well as requests, and for spending caps that stop inference when they are reached. An alert alone arrives after the money has gone. Stolen keys are the other route: in 2024 Sysdig documented attackers using stolen cloud credentials to reach victims’ hosted models, with the victim paying for the usage.

  1. Per-user and per-tenant quotas in tokens per minute and per day, a maximum output length, and a cap on agent steps and tool calls per request.
  2. A hard spending cap on every key and account.
  3. Injection tests for every channel untrusted text can arrive through, scored on whether a forbidden action executed. How to evaluate AI agents describes the method.
  4. Authorisation tests: user A asks for user B’s data through every tool and through retrieval.
  5. Rendering tests in every client, confirming that markup, links and images in model output are neutralised.
  6. A switch that disables the feature, or a single tool, without a deployment.
  7. Provider terms checked: retention settings, training use and the data-processing agreement.

Run: monitoring and incident response

  1. Record every model call and tool call with user, session, model version and prompt version, in a store the application cannot edit.
  2. Alert on spend per user and per key against a baseline, bursts of record lookups, authorisation failures from tools, and a first-ever recipient or destination.
  3. Rerun the security tests whenever the model, a prompt or a tool changes. A provider’s model update counts as a change.
  4. Review tools and permissions every quarter and remove what is unused. A tool trialled in development and never removed is one of OWASP’s own examples of excessive agency.
  5. Keep a short playbook for three incidents: a leaked key, data shown to the wrong person, and an action that should not have happened. For each, write down who can revoke what, and how you would establish what the model read and did.
  6. Practise once. The first rehearsal usually shows that nobody knows how long a key rotation takes.

The logging and key management here are ordinary cloud security work.

Where do teams go wrong?

Three patterns account for most of what we find in reviews.

The access rule lives in the prompt. Someone wrote ‘never reveal other customers’ data’ and considered the matter closed. It holds until the first determined user, and then there is nothing behind it.

The logs were forgotten. The database is locked down and audited, while a tracing tool with forty user accounts holds every prompt in plain text for a year.

The approval step exists and nobody reads it. A reviewer who sees sixty confirmations a day approves all of them. Gate fewer actions and show the raw details, not the model’s summary of them.

How to start this week

For a feature that already exists:

  1. Hold the one-hour threat model with the four questions above.
  2. Search the codebase for every place model output is rendered, executed or stored, and fix those sinks.
  3. Move every access check out of prompts and into tools.
  4. Put a hard spending cap and a per-user quota on the model keys.
  5. Turn on tracing, with redaction.

In our experience steps 2 and 3 remove most of the serious exposure, and neither needs new tooling. This review is how we begin AI agent and generative AI projects at WeStack AI.

Questions

Asked often.

What are the main security risks in an LLM application?

OWASP's 2026 Top 10 for LLM Applications ranks prompt injection first, sensitive information disclosure second and excessive agency third, followed by supply chain, data and model poisoning, unbounded consumption, misinformation, hidden context exposure, vector and embedding weaknesses and improper output handling. Serious incidents usually chain several: an injection, too much permission and an output nobody checked.

Is the OWASP Top 10 for LLM Applications enough for an AI agent?

No. OWASP says that list covers the model as a component inside an application. Once the model calls tools, keeps memory between sessions and takes actions with consequences, the separate Top 10 for Agentic Applications applies as well. Use the first for the application around the model and the second for what the agent is able to do.

Does an LLM feature need its own security review?

Yes, though a short one. Most of the work is ordinary application security: authentication, authorisation, input validation, secrets and logging. The part specific to LLMs is a threat model that treats model output as untrusted, lists every source of text the model reads, and asks what each tool could do if the model were manipulated.

What is denial of wallet?

Denial of wallet is an attack on your bill instead of your uptime. Model calls are charged per token, so anyone who can trigger many expensive requests, or who steals an API key, can run up large costs while the service stays online. The defences are per-user token quotas, hard spending caps on every key and alerts on unusual usage.

Not on the list? Ask us directly

Keep reading

More notes from the build.

All blogs

Let’s build your stack.

Send a brief and we’ll reply with next steps.