28
Twenty Eight Labs

LLM Security

New LLM Attack Vectors and Defensive Patterns

A detailed look at indirect prompt injection, system prompt poisoning, excessive agency, RAG poisoning, tool-call abuse, and practical engineering controls.

Treat every retrieved document, web page, email, and tool response as untrusted input.
Do not rely on the model to enforce security boundaries against itself.
Constrain tools, validate outputs, and isolate secrets outside prompt context.

New LLM Attack Vectors and Defensive Patterns

Published: June 2026 Author: Twenty Eight Labs

Overview

LLM security has moved beyond simple jailbreak prompts. Modern AI products combine user input, retrieved documents, web browsing, tool execution, memory, plugins, code interpreters, and human approval flows. That creates a broad attack surface where the model is only one component of the system.

The most important shift is that attackers no longer need to talk directly to the model. They can place malicious instructions inside documents, emails, tickets, websites, commits, PDFs, calendar events, API responses, or knowledge base entries that the model later consumes.

Indirect Prompt Injection

Indirect prompt injection occurs when malicious instructions are embedded in content the model retrieves or summarizes. The user may never see the hostile instruction, but the model processes it as part of context.

Common carriers include:

  • Public web pages consumed by browsing agents
  • Customer support tickets and CRM notes
  • Uploaded PDFs, spreadsheets, and documents
  • GitHub issues, pull requests, and code comments
  • Email messages processed by enterprise copilots

Defensive Approach

  • Label retrieved content as untrusted data, not instructions
  • Keep privileged system instructions outside retrieved context
  • Strip or quarantine instruction-like text from untrusted documents
  • Require human confirmation before the model performs external actions
  • Log source documents and exact context used for decisions

System Prompt Poisoning

System prompt poisoning targets the configuration or prompt supply chain rather than a single user turn. If an attacker can alter prompt templates, memory, policy files, hidden examples, or agent rules, every future model interaction can inherit the compromise.

This is more persistent than ordinary prompt injection because the malicious behavior becomes part of the product's operating state.

Defensive Approach

  • Version-control prompt templates and policy text
  • Require review for prompt, memory, and agent-rule changes
  • Separate production prompts from user-editable content
  • Monitor prompts for high-risk instructions and secret references
  • Add output filtering outside the model boundary

RAG Poisoning and Retrieval Abuse

Retrieval-augmented generation can be poisoned when the source corpus contains attacker-controlled documents or when ranking logic favors hostile content. The attacker does not need to break the model; they only need to make the wrong context look relevant.

Risk increases when retrieval content can influence actions, citations, account decisions, or code generation.

Defensive Approach

  • Track provenance, ownership, and trust level for every retrieved chunk
  • Prefer allowlisted sources for sensitive workflows
  • Down-rank untrusted content when privileged actions are possible
  • Treat retrieved instructions as evidence, not authority
  • Test with malicious documents that contain hidden or conflicting instructions

Excessive Agency and Tool Abuse

The highest-impact LLM failures often happen after the model has access to tools. A confused model that can only answer text is less dangerous than a confused model that can send email, modify data, deploy code, change permissions, or call internal APIs.

Defensive Approach

  • Split read tools from write tools
  • Use least-privilege tokens scoped to a single workflow
  • Require confirmation for destructive, financial, external, or privilege-changing actions
  • Validate tool arguments in application code
  • Record model reasoning, selected tool, arguments, and final outcome

Output Handling Failures

Model output can become input to browsers, shells, APIs, databases, or other models. If output is rendered or executed without validation, the LLM becomes part of a classic injection chain.

Examples include generated HTML, SQL, shell commands, JSON tool arguments, Markdown links, and code patches.

Defensive Approach

  • Validate structured output against strict schemas
  • Escape generated HTML and Markdown before rendering
  • Never execute generated commands without review
  • Use deterministic policy checks outside the model
  • Treat model output as untrusted until validated

Practical Testing Scenarios

  • Place malicious instructions inside a PDF and ask the model to summarize it
  • Add hidden prompt text to a web page used by a browsing agent
  • Insert a fake policy section into a knowledge base article
  • Ask the model to call a tool using attacker-provided arguments
  • Try to leak system prompts through translation, formatting, or debugging requests
  • Test whether output filters catch secrets before the user receives them

The goal is not to prove that every prompt can be blocked. The goal is to ensure a compromised model cannot cross important system boundaries.