)
How do you add guardrails to LiteLLM?
LiteLLM proxy supports guardrails via a guardrails section in your config.yaml. It integrates with more than 40 third-party guardrail providers covering prompt injection detection, PII masking, content filtering, and policy enforcement.
Each guardrail runs in pre_call, during_call, or post_call mode and can be toggled per-request, per-API-key, or per-team. For enterprise deployments, the native guardrail ecosystem covers the basics but leaves gaps in intent-based policy enforcement, jurisdiction-aware routing, inference-level audit trails, and agent security.
NeuralTrust TrustGuard integrates as a custom guardrail directly in LiteLLM to address those gaps.
TL;DR - Key Takeaways
- LiteLLM is an excellent AI gateway for routing, cost management, and load balancing across 100+ LLM providers. It was not designed as a security product.
- LiteLLM's guardrail ecosystem supports more than 40 providers covering prompt injection detection, PII masking, content filtering, and policy enforcement, each configured with a few lines in
config.yaml. - Guardrails run in three modes:
pre_call(before the LLM call),during_call(in parallel), andpost_call(after the response). - The enterprise gaps: no intent-based policy enforcement, no jurisdiction-aware routing, no inference-level audit trails, no agent security, no adversarial pre-deployment testing.
- NeuralTrust TrustGuard integrates with LiteLLM as a custom guardrail: every application already pointing at the proxy is covered, with no client changes.
- If you want to replace LiteLLM entirely rather than extend it, TrustGate is the alternative. The two products are not designed to sit in series.
LiteLLM routes your AI requests beautifully. It integrates with a large ecosystem of guardrail providers for prompt injection, PII masking, and content filtering. That handles the basics.
For anything enterprise (compliance audit trails, jurisdiction routing, multi-agent monitoring, red teaming) you need a security layer on top. TrustGuard integrates directly into LiteLLM as a custom guardrail, covering input and output on every request without touching your application code.
)
LiteLLM Is Great at Routing. That Is Not the Same as Security.
Picture this. You're three months into a production AI deployment. LiteLLM is working perfectly. It's routing customer queries to GPT-4o, falling back to Claude when OpenAI is slow, tracking spend per team, enforcing rate limits. Your engineers love it.
Then your security lead asks a question: "What's inspecting what goes into those models before it gets there?"
Silence.
LiteLLM was built to solve the multi-provider routing problem. It solved it brilliantly. The GitHub repository has tens of thousands of stars for good reason. But routing AI traffic and securing AI traffic are two different jobs.
This article shows you both: how LiteLLM's guardrail system works, where it stops being enough, and how to integrate NeuralTrust TrustGuard to cover the gaps.
What LiteLLM's Guardrail Ecosystem Gives You
LiteLLM proxy has a guardrails section in its config.yaml. It supports more than 40 guardrail providers, covering four main capability areas:
- Prompt injection detection: scans user inputs for attempts to hijack the model's instructions.
- PII and PHI masking: detects and masks personal data (credit cards, email addresses, SSNs, phone numbers) before it reaches the LLM.
- Secret detection: catches API keys and credentials in inputs.
- Content policy enforcement: flags or blocks non-compliant inputs and outputs against defined policies.
Each guardrail can run in three modes:
pre_call: runs before the LLM call, on input onlyduring_call: runs in parallel with the LLM call, on inputpost_call: runs after the LLM call, on input and output
The basic config structure looks like this:
Start the proxy:
You can toggle guardrails per-request using the metadata.guardrails field:
And enforce specific guardrails at the API key level, useful for giving internal tools different access than customer-facing endpoints:
This is solid for prototyping and most internal deployments. The problem shows up when you need to prove it's working to a regulator.
)
What Native LiteLLM Guardrails Cannot Do
Here is the gap list. These are not feature requests, they are the things that come up in real enterprise deployments.
1. No intent-based policy enforcement
Native guardrail providers detect patterns: known PII formats, known injection signatures. They do not understand what a conversation is trying to accomplish at the semantic level. A user slowly probing for system prompt contents across ten innocent-looking messages will not be caught by pattern-matching alone.
2. No jurisdiction-aware routing
LiteLLM routes by model performance, cost, and load. It does not know that EU personal data should not be routed to a US provider without explicit legal cover. It will happily send a French customer's medical data to an endpoint that exposes you to CLOUD Act jurisdiction.
3. No inference-level audit trail
LiteLLM logs via callbacks: you can send data to Langfuse, Helicone, or S3. That is application-level logging. An enterprise audit trail records the exact input, output, model version, and timestamp at the inference layer in an immutable format a regulator can verify. These are different things.
4. No multi-agent or MCP security
As your LiteLLM deployment evolves from single prompts to agentic workflows (chains of tool calls, memory access, orchestrator-to-subagent communication) native guardrails only see the outermost API call. What happens inside the chain is invisible to them.
5. No adversarial pre-deployment testing
You can configure guardrails for patterns you know about. You cannot know what you do not know. OWASP LLM Top 10 lists ten attack categories. Native guardrails partially cover a few of them. The rest require deliberate red teaming.
6. No compliance documentation
DORA, HIPAA, and the EU AI Act all require documented evidence that your AI system has controls in place. LiteLLM generates logs. It does not generate the audit artifacts a compliance team can present to a regulator.
Adding TrustGuard as a LiteLLM Custom Guardrail
TrustGuard integrates with LiteLLM as a custom guardrail class. Every application already pointing at the proxy is covered with no client changes, you add one Python file and a few lines to your config.yaml.
The guardrail calls TrustGuard's /v1/evaluate endpoint on input before the model call and on the response before it reaches the user. TrustGuard returns a verdict (allow, block, or transform) and the guardrail enforces it.
Step 1: Create the guardrail class
Create a file called trustguard_guardrail.py next to your config.yaml. It uses the httpx client that ships inside LiteLLM, so the proxy image needs no additional dependency:
Step 2: Declare it in config.yaml
Set TRUSTGUARD_API_BASE to {your-workspace-url}/v1/evaluate and inject TRUSTGUARD_API_KEY from your secret manager. The trustguard_guardrail.py file needs to sit next to config.yaml in the directory the proxy runs from.
Step 3: Choose fail-open or fail-closed
fail_open decides what happens when TrustGuard cannot be reached: a timeout, DNS failure, or an error.
fail_open: false (recommended): the proxy returns 503 and the request never reaches the model. Prompts stay inside your perimeter.
fail_open: true: traffic flows uninspected and a warning is logged. Use only if availability is more important than guaranteed inspection, and alert on the failing open (traffic NOT inspected) log line.
Step 4: Choose inspection scope
scope: current_turn inspects only the newest user message plus any tool results since it. Constant payload size; an attack spread across several turns is not visible to a single call.
scope: transcript inspects every message on every turn. Maximum coverage, but payload grows with the conversation and one flagged string in history blocks every later request in that session.
Step 5: Verify
Set LITELLM_LOG=INFO first, then test:
A successful setup logs two lines per request: TrustGuard input -> status=allow and TrustGuard output -> status=allow. A blocked prompt logs status=block on input, and no output line, the model is never called.
Note on streaming: With
stream: true, the post-call guardrail runs on the assembled response after chunks have already been sent. Output-side blocking becomes detection after the fact. Input-side enforcement is unaffected and still runs before the model is called.
See the full integration guide at docs.neuraltrust.ai/trustguard/integrations/litellm.
)
What about TrustGate?
TrustGate is NeuralTrust's AI gateway, a full alternative to LiteLLM, not a layer that sits in front of it. Running two gateways in series adds latency and operational complexity with no benefit.
If you are already running LiteLLM and want to add enterprise security: use TrustGuard as a custom guardrail.
If you are evaluating AI gateways and want native security built in from the start: evaluate TrustGate as your gateway.
Related article: NeuralTurst vs. LiteLLM: AI Gateway Comparison 2026
LiteLLM with TrustGuard vs LiteLLM Alone
| Capability | LiteLLM Alone | LiteLLM + TrustGuard |
|---|---|---|
| Prompt injection detection | Via ecosystem providers (external API dependencies) | Intent-based analysis via TrustGuard evaluate API |
| PII masking | Via ecosystem providers (external server dependencies) | Built-in DLP transform rules, no extra infrastructure |
| Content policy enforcement | Via ecosystem providers | Semantic policy engine with allow, block, transform verdicts |
| Jurisdiction-aware routing | Not supported | Routes by data classification and legal jurisdiction |
| Inference-level audit trail | Callback-based application logs | Immutable inference records per request with trace ID |
| Multi-agent and MCP security | Outermost API call only | Full chain visibility via TrustLens |
| Pre-deployment adversarial testing | Not supported | TrustTest covers OWASP LLM Top 10 |
| Compliance documentation | Not included | DORA, EU AI Act, HIPAA audit artifacts |
| Per-team, per-key guardrail control | Yes (LiteLLM native) | Yes, plus data classification controls |
| Application code changes required | N/A | None, proxy-level integration |
What This Means in Practice
LiteLLM handles the routing problem. TrustGuard handles the security problem. The two work at different layers and are designed to run together.
TrustGuard adds runtime enforcement on every request: input evaluation before the model call, output evaluation before the response reaches the user, immutable audit records for every interaction, and anomaly detection for unusual access patterns. TrustLens maps every AI touchpoint across your environment. TrustTest runs adversarial exercises before you go live.
The combination covers both halves of what enterprise AI infrastructure requires: routing that works and security you can prove.
Frequently Asked Questions
1. How do I add guardrails to LiteLLM?
Add a guardrails section to your LiteLLM config.yaml. Each entry needs a guardrail_name, a litellm_params block specifying the guardrail provider and a mode (pre_call, during_call, or post_call). Set default_on: true to apply the guardrail to all requests. LiteLLM supports more than 40 guardrail providers, including custom Python classes for enterprise integrations. Full documentation is at docs.litellm.ai/docs/guardrail_providers.
2. What does LiteLLM's guardrail ecosystem cover?
LiteLLM's guardrail ecosystem spans more than 40 providers across four main capability areas: prompt injection detection, PII and PHI masking, content policy enforcement, and secret detection. Guardrails run on input before the model call (pre_call), in parallel during the call (during_call), or on the response after the model replies (post_call). Each guardrail is an external API call, so the enterprise security guarantee depends entirely on the reliability and coverage of the provider you choose.
3. What security gaps does LiteLLM leave for enterprise deployments?
LiteLLM's guardrail ecosystem does not provide intent-based policy enforcement (only pattern matching from external providers), jurisdiction-aware routing, inference-level audit trails in compliance-ready formats, multi-agent workflow security, or pre-deployment adversarial testing. For organisations subject to DORA, HIPAA, the EU AI Act, or FedRAMP, these gaps need to be closed by an integrated security layer like NeuralTrust TrustGuard.
4. How does NeuralTrust TrustGuard integrate with LiteLLM?
TrustGuard integrates with LiteLLM as a custom guardrail class. You create a trustguard_guardrail.py file next to your config.yaml, declare it in the guardrails section, and set two environment variables: TRUSTGUARD_API_BASE (pointing to /v1/evaluate on your TrustGuard workspace URL) and TRUSTGUARD_API_KEY. The guardrail runs pre_call and post_call on every request. No changes to your application code are required. Full instructions are at docs.neuraltrust.ai/trustguard/integrations/litellm.
5. What is the difference between fail-open and fail-closed in TrustGuard?
fail_open: false (fail-closed) means that if TrustGuard is unreachable, the proxy returns a 503 error and the request never reaches the model, prompts stay inside your perimeter. fail_open: true means that if TrustGuard is unreachable, traffic flows uninspected and a warning is logged. Fail-closed is the recommended default for any deployment where security is a hard requirement. If you use fail-open for availability reasons, set an alert on the failing open (traffic NOT inspected) log line so an outage does not go unnoticed.
About the Author
Roger Howroyd is Head of Global SEO and AI at NeuralTrust, where he leads the company's search strategy across SEO, AEO, GEO, and LLM optimization. He specializes in AI-powered search, content strategy, backlink development, and SEM. Connect on LinkedIn.
NeuralTrust is an AI agent security platform, recognized in the Gartner Hype Cycle for Application Security 2026, the Gartner Hype Cycle for Infrastructure Security 2026, the Gartner 2025 Market Guide for AI Gateways and Guardian Agents, and the KuppingerCole 2025 Leadership Compass for Generative AI Defense. ISO 27001 certified. Headquartered in Barcelona.
)
)
)
)