)
Last updated: November 10, 2026
A data scientist loads a popular open-weight model with one line of Python. Before the first prediction, the file has already run someone else's code. What does AI model security have to cover to stop that? It covers every stage a model passes through: where it comes from, what it was trained on, where its weights live, and how it behaves in production.
TL;DR: Key Takeaways
- Malicious models add up: researchers curated 18,195 malicious pre-trained models on Hugging Face, which stayed undetected for over 233 days on average (Zhang et al., 2026).
- Unsafe formats persist: 9.3% of 10,023 popular Hugging Face repositories created in the past year still rely on Pickle (Qin et al., 2026).
- Scanners miss things: one new attack family evades ten model scanners, with a 63% evasion rate for its strongest variant (Pradhan et al., 2026).
- Poisoning is cheap: as few as 250 malicious documents backdoored models from 600 million to 13 billion parameters in a controlled study (Anthropic, 2025).
- Access is the gap: only 40% of organizations use access controls on AI models and data, and 92% of those with an AI-related breach lacked them (IBM, 2026).
At a glance: AI model security across the lifecycle
Quick Answer
What does AI model security include?
AI model security protects a model at four points: the files you download, the data that shaped it, the weights you store, and the outputs it produces in production. Each point needs its own control. Scanning and signing cover files, data review covers training, access control covers weights, and runtime inspection covers behavior.
| Stage | Main threat | Example evidence | First control |
|---|---|---|---|
| Model files | Code that runs when a file loads | 18,195 malicious models on one hub (2026) | Safe formats, scanning, signature checks |
| Training data | Poisoning and hidden backdoors | 250 documents enough in a lab setting (2025) | Data provenance and adversarial tests |
| Weights | Theft and tampering | 38 distinct attack vectors mapped (2024) | Few copies, few people, monitored access |
| Runtime | Prompt injection, data leakage, misuse | 92% of AI-breached firms lacked access controls (2026) | Inspection of prompts, outputs, and tool calls |
How we compared
We read two 2026 studies of malicious Hugging Face models, a 2026 scanner evasion paper, and Anthropic's 2025 poisoning study. We added RAND, OWASP LLM03:2025, OpenSSF, and IBM's 2026 breach data.
Where model files turn into code
Many model formats are not just data. Pickle, the default behind many PyTorch checkpoints, stores instructions that Python executes while loading. A malicious file can open a reverse shell or read credentials before your code touches a single weight. That makes model files part of your software supply chain.
According to Zhang et al. (2026), more than 81% of the malicious models they studied combined several tactics, such as metadata errors and unsafe serialization. The four detection tools they tested showed high false negative rates.
Safer formats exist, but adoption is uneven. Qin et al. (2026) found 9,544 Pickle-based models across 929 popular repositories created between July 2025 and June 2026, some from major publishers.
| Format | Can run code on load? | Typical use | What to do |
|---|---|---|---|
| Pickle (.pkl, .pt, .bin, .ckpt) | Yes | Many PyTorch checkpoints | Avoid, or scan and load in a sandbox |
| Safetensors | No, stores tensors only | Many open-weight models | Prefer it, still verify the source |
| GGUF | No, schema-based data | Local LLM runtimes | Prefer it, still verify the source |
Treat scanners as one layer. Pradhan et al. (2026) showed attacks that overwrite trusted Python modules to slip past allowlists. ReversingLabs (2025) found two 7z-packed malicious models the hub's Pickle scanner did not flag; Hugging Face removed them within 24 hours.
)
Poisoned data and hidden backdoors
Poisoning changes what a model learns instead of what its file does. An attacker plants examples in training or fine-tuning data so the model behaves normally until a trigger appears. Poisoning is hard to spot with file scanning, because the weights look like any other weights. Testing behavior is the main way to find it.
The cost of an attack may be lower than teams assume. Anthropic (2025) found that 250 poisoned documents backdoored every model size it tested; for the largest model, that was about 0.00016% of training tokens. The study used a narrow, low-stakes backdoor that made models output gibberish. The authors say it is unclear whether the pattern holds for larger models or more harmful behaviors.
Fine-tuning adds a closer risk for most enterprises. OWASP (2025) warns that a malicious LoRA adapter can compromise a trusted base model, and that model cards offer no guarantee of origin. Treat adapters and fine-tuning datasets as untrusted until checked.
Protecting model weights from theft
Weights are the asset. Whoever holds them can run, copy, or modify the model without your controls. Protecting them means limiting where copies live, who can reach them, and how they leave. RAND's research shows there is no single fix, so plan for several layers.
According to RAND (2024), securing frontier model weights involves 38 distinct attack vectors. It considers attackers from opportunistic criminals to nation-state operations and defines five security levels. Its core advice works for any valuable model: centralize copies on a few monitored systems, reduce who can access them, harden interfaces against exfiltration, and run an insider threat program.
Most organizations are not there yet. According to IBM (2026), only 40% use access controls on AI models and data. For theft through the API, our guide to AI model theft covers extraction and infrastructure controls.
AI model security controls that work
Effective AI model security combines prevention before deployment with verification at every handoff. Prefer formats that cannot execute code, scan what you cannot avoid, and sign models so tampering shows. Then test behavior with red teaming, because file checks cannot reveal a poisoned model or a weak safety tune.
| Control | Addresses | Where it runs | Limitation |
|---|---|---|---|
| Safe formats (Safetensors, GGUF) | Code execution on load | Model selection | Custom loading code can still run |
| Model scanning | Known malicious patterns in files | CI/CD and model registry | Evasion techniques keep emerging |
| Model signing and hashes | Tampering after training | Upload, deployment, loading | Proves origin, not good behavior |
| Red teaming | Backdoors, unsafe behavior, jailbreaks | Before launch and after changes | Covers the cases you test |
| Runtime inspection | Prompt injection and data leakage | Production traffic | Needs tuning to cut false alerts |
Signing deserves more use. According to OpenSSF (2025), its model-signing project supports any model format and size and several signing methods, including Sigstore. It recommends signing during training and verifying at upload, deployment, and use.
OWASP (2025) advises using only models from verifiable sources, with signing and file hashes, plus AI bills of materials such as OWASP CycloneDX. For attacks on model behavior, see our guide to securing LLMs from adversarial attacks.
Security and governance: models in production
A verified model can still be misused once it is live. IBM (2026) found that AI-related breaches rose to 21% of the breached organizations it studied, from 13% a year earlier. Breaches involving a malicious model averaged $4.94 million. Those involving model inversion, which extracts sensitive data from a model, averaged $6.07 million.
Three production risks deserve the most attention:
- Prompt injection: crafted inputs, including hidden text in documents, override instructions.
- Data leakage: models return sensitive records or system prompts.
- Tool misuse: a manipulated model triggers actions it should never run.
IBM says the most common causes of AI-related breaches were not model failures but weaknesses in surrounding systems. Our overview of AI security risks in 2026 covers the wider threat list.
Model files and training data need the vendor-neutral controls above. In production, you secure models on NeuralTrust. Every model call routes through Agent Gateway (TrustGate), one place you control and meter. You inspect every interaction with Agent Runtime Security (TrustGuard) and test models with AI Red Teaming (TrustTest).
Which should you choose?
Your priorities depend on how you get and use models. Teams pulling open-weight models need file security first. Teams fine-tuning need data and adapter controls. Teams serving valuable proprietary models need weight protection. Almost everyone serving models in production needs runtime inspection. The table matches common situations to a first step.
| If you... | Start with | Why |
|---|---|---|
| Download open-weight models from public hubs | Safe formats, scanning, and signature checks | Model files can execute code on load |
| Fine-tune with external data or adapters | Data provenance and red teaming after each tune | Poisoning and malicious adapters hide in behavior |
| Serve a proprietary model | Few weight copies, least-privilege access, monitoring | Weights are the asset attackers want |
| Call third-party model APIs only | Runtime inspection and access control | Your risk sits in prompts, outputs, and tools |
Conclusion
AI model security starts before deployment and never really stops. Malicious files, poisoned data, stolen weights, and runtime attacks each need a different control, and no single scanner covers all four. Prefer safe formats, sign and verify every handoff, keep weights on few systems, and red team after every change. Then inspect what models do in production, where IBM traces the most common breach causes to surrounding systems.
Secure AI Models in Production with NeuralTrust
Verify model files before deployment, then route every model call through one clear layer that inspects each prompt and output.
)
Related Comparisons
- AI Model Theft: How to Prevent It
- Why Your AI Model Might Be Leaking Sensitive Data
- AI Agent Security: Distillation Attacks
- AI Hacked Hugging Face Twice: CISO Guide
FAQs about AI Model Security
1. What is AI model security?
AI model security is the set of controls that protect a machine learning model across its lifecycle. It covers the files you download, the training and fine-tuning data, the stored weights, and production behavior. It works alongside application and data security, because IBM traces the most common causes of AI-related breaches to surrounding systems.
2. What are the main AI model security risks?
The main risks are malicious model files that run code on load, poisoned training data, and theft or tampering of weights. Runtime attacks such as prompt injection and data leakage complete the list. IBM's 2026 report found that breaches involving a malicious model averaged $4.94 million, while model inversion breaches averaged $6.07 million.
3. What is AI model scanning?
AI model scanning inspects model files before you load them. Scanners look for unsafe deserialization, dangerous imports, and embedded code. They work best in CI/CD and model registries, so nothing reaches production unchecked. Research in 2026 showed new attacks can evade many scanners, so combine scanning with safe formats and signing.
4. How do you test AI model security?
Test files, behavior, and deployment separately. Scan and verify signatures on every file. Red team the model for jailbreaks, prompt injection, data leakage, and backdoors, especially after fine-tuning. Then test access controls and tool permissions. OWASP recommends extensive red teaming whenever you select a third-party model.
5. Are Safetensors files safe?
Safetensors files store tensors only, so loading one does not execute embedded code the way a Pickle file can. That removes the most common file-level attack. It does not prove the model is trustworthy. A Safetensors model can still be poisoned, backdoored, or published by an impostor, so verify the source and test behavior.
6. What is model signing?
Model signing attaches a cryptographic signature to model files so anyone can verify who produced them and that nothing changed since. The OpenSSF model-signing project reached version 1.0 in April 2025. It supports any model format and size, and works with Sigstore, self-signed certificates, or public and private key pairs.
About the Author
Roger Howroyd is Head of Global SEO and AI at NeuralTrust, where he leads the company's search strategy across SEO, AEO, GEO, and LLM optimization. He specializes in AI-powered search, content strategy, and SEM. Connect on LinkedIn.
NeuralTrust is the leading platform for securing and scaling AI agents. Named a Pioneer in the Gartner Emerging Market Quadrant for AI Application Security 2026, recognized across four Gartner Hype Cycle reports in 2026, and featured in the Gartner Market Guide for Guardian Agents 2026, the Gartner Market Guide for AI Gateways 2025 and the KuppingerCole Leadership Compass for Generative AI Defense 2025. Headquartered in Barcelona with offices in London and New York. ISO 27001 certified.
Sources
- Zhang, Hu, Xia, Lo, and Li, "A Closer Look at the Malicious Pre-Trained Models on Hugging Face," ICSE 2026 Journal Ahead Workshop, April 13, 2026. https://conf.researchr.org/details/icse-2026/jaws-2026-papers/33/A-Closer-Look-at-the-Malicious-Pre-Trained-Models-on-Hugging-Face
- Qin et al., "DITTO: A Context-aware Pickle-based Pre-Trained Model Scanner for Effective Security Audits," arXiv, October 2026. https://arxiv.org/abs/2610.10735
- Pradhan, Nambiar, and Soremekun, "ShadowPickle: Evading Machine Learning Model Scanners via Stealthy Pickle Deserialization Attacks," arXiv, July 2026. https://arxiv.org/abs/2607.17503
- ReversingLabs, "Malicious ML models discovered on Hugging Face platform," February 6, 2025. https://www.reversinglabs.com/blog/rl-identifies-malware-ml-model-hosted-on-hugging-face
- Anthropic, UK AI Security Institute, and The Alan Turing Institute, "A small number of samples can poison LLMs of any size," October 9, 2025. https://www.anthropic.com/research/small-samples-poison
- RAND, "Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models," May 30, 2024. https://www.rand.org/pubs/research_reports/RRA2849-1.html
- OWASP GenAI Security Project, "LLM03:2025 Supply Chain," 2025. https://genai.owasp.org/llmrisk/llm032025-supply-chain/
- OpenSSF, "Launch of Model Signing v1.0," April 4, 2025. https://openssf.org/blog/2025/04/04/launch-of-model-signing-v1-0-openssf-ai-ml-working-group-secures-the-machine-learning-supply-chain/
- IBM and Ponemon Institute, "Cost of a Data Breach Report 2026," July 29, 2026. https://assets.ibm.com/adobe/assets/urn:aaid:aem:21111142-1251-4369-86fb-57b82f5bb108/original/as/Cost%20of%20a%20Data%20Breach%20Report%202026.pdf
)
)
)
)