Detect vulnerabilities in LLM applications with Datadog’s AI-native SAST

AI coding tools help developers build and deploy LLM applications quickly, but this speed comes with new security risks. Traditional static application security testing (SAST) tools that are pattern based weren’t designed to detect LLM-specific issues such as prompt injection sinks and exposed system prompts. These vulnerabilities often don’t become apparent until applications are already running in production, when remediation is more difficult and expensive.

Datadog Code Security’s AI-native SAST offers coverage for the OWASP Top 10 for LLM Applications to help you detect LLM-specific risks earlier in the development cycle. Developers get actionable feedback directly in the same workflows where they review and ship code, helping them remediate vulnerabilities faster.

In this post, we’ll explain: 

• What the OWASP Top 10 for LLM Applications covers

• How AI-native SAST provides detection

Close dialog

What is the OWASP Top 10 for LLM Applications?

The OWASP Top 10 for LLM Applications is an industry-standard reference that describes vulnerability classes specific to applications that use LLMs. The list is the LLM equivalent of the traditional OWASP Top 10. It covers risks related to how user input reaches a model, how model outputs are handled downstream, whether LLM-driven actions are authorized, and more.

The following table summarizes Datadog Code Security’s AI-native SAST coverage within each risk in the OWASP Top 10 for LLM Applications:

How Datadog’s AI-native SAST detection works

Traditional SAST rules require a trade-off. If you make the rules too broad, they produce noise. If you make them too narrow, they miss real vulnerabilities. 

Datadog Code Security’s AI-native SAST goes beyond rule matching by using LLMs to reason about code context and data flow. It then independently verifies candidate findings. Findings surface directly in the Datadog platform as pull request (PR) comments and through CI checks via PR Gates.

AI-native SAST coverage for the OWASP Top 10 for LLM Applications is available across Python, Go, Java, C#, TypeScript, and JavaScript. The following examples show vulnerable application code and illustrate the detection approaches for three of these risks: 

• Taint analysis for prompt injection

• Control flow analysis for excessive agency

• Pattern matching for hidden context exposure

Taint analysis detection for prompt injection

Prompt injection (LLM01) occurs when external input is injected directly into an LLM prompt without sanitization, allowing the model to treat attacker-controlled data as an instruction rather than content. An attacker can use this method to bypass safety guardrails, exfiltrate data, or manipulate the model into taking actions that the model was never intended to take.

AI-native SAST uses taint analysis to identify paths where untrusted input reaches an LLM call. Consider a Go handler that reads a query parameter and concatenates it directly into a prompt:

1
func askHandler(w http.ResponseWriter, r *http.Request){
2
    // User-controlled input pulled directly from the request
3
    userQuery := r.URL.Query().Get("q")
4

5
    // Tainted value concatenated into the prompt without sanitization
6
    response, err := llm.Complete(ctx, "Answer the following: " + userQuery)
7
    if err != nil {
8
        http.Error(w, err.Error(), http.StatusInternalServerError)
9
        return
10
    }
11
    w.Write([]byte(response))
12
}

In this example, userQuery is an untrusted source, and the LLM call is the sink. AI-native SAST can trace the flow between them and flag any path where the value reaches the prompt without appropriate handling or validation. This analysis focuses the finding on the data flow that creates the risk instead of matching only the presence of an LLM call or user input.

Control flow analysis detection for excessive agency

Excessive agency (LLM03) occurs when an LLM takes consequential actions without having the appropriate authorization checks in place. Such actions include executing code, calling external APIs, and modifying data.

Detecting excessive agency requires control flow analysis because AI-native SAST needs to determine what capabilities the LLM is granted before it can act. Consider this Python example, where an LLM-powered agent is initialized with tools that grant unrestricted access to the shell and file system:

1
def create_agent(user: User) -> Agent:
2
      tools = [
3
          ShellTool(),  # Can execute arbitrary commands
4
          FileTool(root="/"),  # Can access the entire file system
5
      ]
6
      return Agent(tools=tools)

In this example, create_agent grants the agent unrestricted shell and file system access regardless of who the user is or what they are authorized to do. AI-native SAST can identify that no capability restrictions or authorization checks are applied before the agent is returned with these tools.

In a larger application, the same agent configuration might be reachable through multiple callers or conditional branches. Control flow analysis enables AI-native SAST to examine those paths and flag cases where at least one path initializes the agent without the required restrictions in place. 

Pattern matching detection for hidden context exposure 

Hidden context exposure (LLM08) is the revealing of sensitive information that is not intended for users. This information can include system prompts, tool schemas, RAG-retrieved content, and developer instructions that are written to logs or returned in API responses. 

AI-native SAST uses pattern matching to identify code that exposes system prompt contents. For example, the following Java method writes its system prompt to an application log:

1
public String createResponse(String userMessage) {
2
    String systemPrompt = "You are an internal assistant. Do not reveal...";
3

4
    // System prompt logged before the LLM call, exposing sensitive instructions
5
    logger.info("Processing request with context: {}", systemPrompt);
6

7
    return llmClient.complete(systemPrompt, userMessage);
8
}

AI coding assistants can introduce this type of exposure by generating debug logging that includes system prompt contents. In this example, systemPrompt does not originate from untrusted input. The value is sensitive because it contains instructions that configure the LLM application’s behavior. AI-native SAST can identify that systemPrompt is passed directly to logger.info(), where its contents could be exposed.

Start detecting LLM vulnerabilities with AI-native SAST

AI-native SAST in Datadog Code Security helps developers identify LLM-specific vulnerabilities before they reach production. Techniques such as taint analysis, control flow analysis, and pattern matching identify risks that require context beyond conventional static rules. To learn more, read the AI-native SAST documentation.

If you don’t have a Datadog account, you can sign up for a 14-day free trial to get started detecting vulnerabilities in your LLM applications.