What is code injection in LLMs?

Code injection in LLMs occurs when user input causes the model to generate or execute harmful commands, such as SQL queries, shell scripts, or API calls. This can lead to unauthorized access or system compromise.

How is code injection different from prompt injection?

Prompt injection manipulates the LLM's behavior or response, while code injection aims to execute malicious commands via downstream systems like databases or interpreters.

What are common code injection attacks in LLMs?

Common types include SQL injection, shell command injection, API tampering, scripting injection, and header-based injection—all targeting how LLMs generate or interact with executable code.

How do attackers exploit LLM code injection?

Attackers craft inputs that appear benign but cause the LLM to generate dangerous outputs. These outputs are executed by connected systems if not properly validated or isolated.

What causes code injection in AI systems?

Code injection often results from over-trusting LLM outputs, merging raw inputs with system logic, or lacking validation between the LLM and execution environments like shells or APIs.

How can I prevent code injection in LLMs?

You can prevent code injection by combining prompt validation, input sanitization, secure prompt design, contextual isolation, and runtime monitoring of LLM interactions.

What is the role of an LLM gateway in security?

An LLM gateway helps detect and block injection attempts by filtering prompts, monitoring context, and enforcing security policies across AI interactions and system integrations.

Why use red teaming for LLM security?

Red teaming helps uncover injection vulnerabilities by simulating real-world attacks, identifying unsafe execution paths before they can be exploited in production.

Back

Code Injection in LLM Applications

Joan Vendrell • March 26, 2025

Contents

Code injection is emerging as one of the most critical security risks in AI-powered systems, particularly when large language models (LLMs) are connected to databases, APIs, or scripting environments. As enterprises increasingly rely on LLMs to automate queries, generate code, or orchestrate system tasks, the attack surface expands, turning prompt inputs into potential vectors for exploitation.

In this post, we break down how code injection works in the context of AI, how it differs from prompt injection, and what defenses you can implement to protect your stack. If you're also exploring LLM security best practices or comparing AI gateways vs. traditional API gateways, our blog will help deepen your understanding of AI-native vulnerabilities.

What Is Code Injection?

Code injection is a form of security exploitation where an attacker tricks an application into executing unintended commands or queries. Traditionally, code injection has been studied in the context of web applications (e.g., SQL injection, shell command injection) where improperly sanitized inputs lead to untrusted code execution.

In LLM-based systems, we see a new flavor of this threat. An LLM might generate or execute an SQL query, an API call or a script based on a user prompt. If the user’s input is malicious, it could manipulate such external system calls, resulting in the execution of harmful commands or queries, unauthorized data access, remote code execution, or privilege escalation.

Code Injection vs. Prompt Injection

While related, “prompt injection” typically involves tricking an LLM into returning specific text or ignoring developer-imposed constraints. Code injection goes a step further: the adversary’s goal is not just to manipulate text, but to run malicious commands or queries within the larger application context.

How Code Injection Affects LLMs and Agents

LLMs and autonomous AI agents play a pivotal role in modern applications, acting as intermediaries between user inputs and system operations. These agents often generate code or dynamic content that is executed by other subsystems, making them prime targets for injection attacks that exploit their ability to interface with databases, APIs, and system commands. Below are common scenarios in which code injection becomes dangerous:

Direct Code Interpretation by LLMs: Some LLM-based environments allow users to run code that the model has generated. If an attacker can embed malicious scripts into the LLM’s output, these scripts might be executed without proper review.
Passing Malicious Parameters to Database Queries: Developers sometimes rely on LLMs to build SQL or NoSQL queries from natural language instructions. An attacker could craft input that forces the LLM to build queries containing DROP TABLE or other destructive commands.
API Calls or System Commands: Systems may allow the LLM to call API endpoints. Attackers can abuse this by injecting additional parameters, paths, or payloads, leading to undesired behavior. Similarly, when LLMs generate shell commands, injection of hostile flags or arguments is possible.
Non-Prompt Injections: It’s not always the prompt that carries malicious code injections. HTTP headers, query parameters, environment variables, or form fields can contain injection strings that ultimately shape the LLM’s actions when interfacing with databases, APIs, and system commands. This is especially relevant when the application merges these values into a single context string for the LLM.

How Code Injection Attacks Work

Attackers generally rely on two ingredients for code injection: a vulnerable “entry point” and a system that will execute or interpret the injected code. In an LLM scenario:

1. Exploiting the LLM’s Natural Language Interface The attacker provides an input that looks like legitimate text but is crafted to yield a malicious payload when the LLM executes queries or commands. 2. Chaining Vulnerabilities LLMs rarely operate in isolation. Code injection becomes possible when the LLM’s output is taken at face value by another component like a database layer, a shell, or a third-party API. If the system does not sanitize or validate the LLM’s output, the attacker can escalate privileges or run arbitrary code. 3. Insecure Execution Paths Code injection often stems from over-trusting LLM outputs. When developers connect LLMs directly to execution environments such as interpreters, CI pipelines, or API clients without introducing review or validation gates, they inadvertently create attack surfaces.

Types of Code Injection Attacks

Code injection attacks come in various forms, each targeting different components of an LLM-based system. These attacks exploit vulnerabilities in the way models generate, interpret, or execute code, leading to unintended consequences such as unauthorized data access, system manipulation, or remote code execution (RCE). Below, we outline the most common types of code injection attacks and their potential impact on LLM applications.

SQL Injection & NoSQL Injection

Attackers inject malicious SQL or NoSQL statements such as DROP TABLE users; or $ne: null into user prompts. If the LLM is responsible for constructing database queries, it may inadvertently pass these statements to the database without sanitization. This can lead to data leaks, table deletion, or unauthorized access to restricted records, especially when developers rely on natural language-to-query generation.

Shell Command Injection

When LLM outputs are interpreted as shell commands or scripts, attackers can embed special characters like |, &&, or ; to chain unauthorized commands. This could enable anything from listing sensitive files to initiating a reverse shell, depending on the privileges of the underlying execution environment. This type of injection is particularly dangerous in DevOps assistants or agent-based systems that automate command-line tasks.

API Parameter Injection

If an LLM generates or manipulates API calls, attackers may craft prompts that alter endpoint URLs, modify HTTP methods, or insert rogue parameters. For example, adding ?admin=true or injecting malicious payloads into JSON bodies can lead to privilege escalation, data tampering, or even denial-of-service conditions. When LLMs are connected to real-world systems, these attacks can trigger destructive or unauthorized behavior.

Scripting Language Injection

LLMs often generate code in languages like Python or JavaScript for use in downstream components. Attackers can abuse this capability by embedding malicious logic into functions, such as infinite loops, data exfiltration scripts, or os.system() calls. If the code is executed without human review, this opens the door to unintended system manipulation and potential RCE vulnerabilities.

Header-Based or Metadata-Based Injection

Malicious code can also be hidden in non-prompt input sources such as HTTP headers, cookies, query parameters, or metadata fields. If these inputs are merged into the context passed to the LLM, the model may unknowingly generate output that reflects or executes the malicious payload. This is especially dangerous in multi-modal systems where user context is constructed from various dynamic sources.

How to Protect LLMs Against Code Injection

Preventing code injection in LLM-powered systems requires a layered approach.** No single control is sufficient**: developers must combine input sanitization, prompt hardening, environment isolation, and runtime monitoring. Centralizing these protections at an LLM gateway is a way to deploy a multi-layered, scalable defense system.

Prompt-Level Code Injection Detection: Use dedicated language models to analyze prompts for signs of malicious or out-of-context code instructions. These detection models can identify suspicious suspicious or hidden code injections within prompts.
Application-Level Injection Filtering: Deploy injection detection systems at the application layer to scan HTTP headers, request parameters, and payloads for known injection patterns. This prevents upstream context poisoning that may affect the LLM's interpretation or decision-making, even if the prompt itself appears benign.
Code Sanitization: Block code inputs entirely in prompts for use cases that do not require code interpretation or execution. If user-supplied code is necessary, sanitize and escape it to prevent downstream components from misinterpreting it as executable. This is especially important in applications where LLM outputs are automatically passed to interpreters, scripts, or other systems.
Prompt Design and Contextual Isolation: User instructions should be isolated from system logic to prevent unintended behaviors. Merging raw user input with commands or queries without validation opens the door to injection. Safer alternatives like parameterized queries or templating help mitigate this risk.
Red Teaming: Run continous adversarial tests on your LLM and agent endpoints to find injection vulnerabilities before attackers do, and identify where code injections might lead to unsafe execution.
Monitoring and Anomaly Detection: Log all prompts and interactions between user and LLM, and set up a real-time alerting system that looks for malicious or unusual patterns, such as repeated code statements.

Conclusion

Code injection is a major risk in AI-powered applications, especially when LLM models interface with external systems such as SQL databases, APIs, and code interpreters. A combination of defensive mechanisms like LLM gateways and offensive strategies such as red teaming, supported by good development practices like input sanitization, can eliminate code injection vulnerabilities in your LLM applications.