)
Last updated: November 10, 2026
Your model's best answers are also someone else's cheapest training data. That is the uncomfortable lesson of 2026's distillation campaigns. What does it mean for AI agent security? A distillation attack queries a model at scale and trains a cheaper copy on the outputs, especially on its reasoning. Frontier labs are the main targets today, but any enterprise that exposes a fine-tuned model or agent through an API inherits the same risk.
TL;DR
- Industrial scale: three labs generated over 16 million exchanges with Claude through about 24,000 fraudulent accounts (vendor-reported; Anthropic, 2026).
- Not just one vendor: Google detected a campaign of more than 100,000 prompts aimed at copying Gemini's reasoning (CSO Online, 2026; GTIG, 2026).
- Defenses can work: Anthropic says new safeguards cut the reasoning tokens distillers extract by 10 to 100 times, with a false positive rate under 0.1% (vendor-reported; Anthropic, 2026).
- Cheap to attempt: researchers extracted the full projection matrix of two production models for under $20 in queries (Carlini et al., 2024).
- Costly when it hits: breaches involving AI model inversion averaged $6.07 million, 18% above the global average (IBM, 2026).
At a glance: distillation vs other model theft
Quick Answer
Is a distillation attack an AI agent security problem?
Yes, whenever you expose a model or agent that others would pay to copy. Distillation uses legitimate access, so firewalls and malware tools rarely see it. It shows up as unusual query patterns, fraudulent accounts, and prompts that fish for hidden reasoning. That makes identity, rate limits, and traffic analytics the first line of defense.
| Attack | Access needed | What gets stolen | Public example | First defense |
|---|---|---|---|---|
| Distillation (functional replication) | Normal API or app access at scale | Capabilities and reasoning style | Claude and Gemini campaigns, 2026 | Account verification and extraction detection |
| Parameter extraction | Black-box API with detailed outputs | Parts of the model's weights | Projection matrix recovery, 2024 | Limit output detail such as logits |
| Weight theft | Access to storage or serving infrastructure | The full model | Insider or infrastructure compromise | Infrastructure security and key control |
| System prompt extraction | Chat access | Instructions and business logic | Common in public chatbots | Prompt hardening and output filtering |
How we compared
We read Anthropic's February and October 2026 reports, Google's February 2026 threat report and its press coverage, and the 2024 model-stealing paper by Carlini and colleagues. We mapped each defense to OWASP's LLM10:2025 guidance. Figures reported by model vendors about their own platforms are labeled vendor-reported.
How a distillation attack works
Distillation trains a small "student" model to imitate a larger "teacher." The legitimate version is a standard machine learning technique. The attack version does it without permission, through networks of fake accounts, and targets the teacher's reasoning traces because they show how an answer was reached, not just the answer.
Attackers usually start with passive extraction. They send normal-looking requests and train on the responses. Anthropic (2026) reported that summarizing Claude's reasoning made transcripts far less useful. A model trained on full transcripts scored 80.4% on SWE-bench Verified, versus 71.8% when trained on summaries (vendor-reported).
When passive collection stops paying off, attackers switch to active extraction. They ask the model to repeat its prior reasoning verbatim, translate it into another language, or write it inside tags next to the answer. According to Anthropic (2026), one lab tested 12,000 requests, each trying a different trick.
Scale is the other ingredient. Commercial proxy services resell access through what Anthropic calls "hydra cluster" architectures. When one account is banned, another takes its place. In one case, a single proxy network managed more than 20,000 fraudulent accounts at once.
)
What the 2026 cases tell us
The 2026 disclosures show distillation is organized, persistent, and fast to adapt. Anthropic attributed three campaigns to named labs with high confidence. Google reported frequent extraction attempts from private companies and researchers worldwide. Both say the activity violates their terms of service.
By October, Anthropic said it had banned tens of thousands of accounts associated with distillation (Anthropic, 2026). According to CSO Online (2026), Google detected and blocked the Gemini campaign; Google itself says it lowered the risk of the attack.
| Campaign (per Anthropic, February 2026) | Exchanges | Main targets |
|---|---|---|
| DeepSeek | Over 150,000 | Reasoning, rubric-based grading, censorship-safe rewrites |
| Moonshot AI | Over 3.4 million | Agentic reasoning, tool use, coding, computer use, vision |
| MiniMax | Over 13 million | Agentic coding, tool use, and orchestration |
Look at the targets. Anthropic said each campaign went after Claude's most differentiated capabilities: agentic reasoning, tool use, and coding. Those are exactly the skills enterprises build their agents on.
Speed is the next lesson. When Anthropic released a new model during MiniMax's active campaign, MiniMax redirected nearly half its traffic to the new system within 24 hours. A defense that takes a quarter to deploy is already late.
Google's case adds a language angle. According to CSO Online (2026), attackers told Gemini to keep its "thinking content" in the same language as the user input. Google said the breadth of questions suggested an attempt to copy Gemini's reasoning in non-English languages (GTIG, 2026).
Why AI agent security teams should care
AI agent security usually focuses on what agents do to you: prompt injection, data leakage, tool misuse. Distillation flips the direction. It is about what others take from your agents. If your agent wraps a fine-tuned model, proprietary workflows, or expert reasoning, its outputs are an asset someone can harvest.
Google's report makes the point directly. According to CSO Online (2026), Google said model extraction should worry any organization that provides AI models as services, and advised monitoring API access for signs of systematic extraction.
Agents raise the stakes in three ways. They often expose step-by-step plans, which look a lot like the reasoning traces distillers want. They run on usage-based APIs, where a fraudulent account looks like a good customer. And they sit behind partner integrations, where traffic from a reseller can hide a proxy network.
The governance gap makes this harder. Gartner (2026) found that 71% of organizations expect to deploy agentic AI within a year, while only 21% have agentic AI governance capabilities. Few teams are watching their agents' outbound value at all.
Distillation also affects agents you buy, not only agents you build. Models distilled without permission may ship without matching safeguards. Anthropic (2026) argued that illicitly distilled models have in practice been released without proportionally strong safeguards. If your team adopts a cheaper model, red team it before it touches production tools.
Signals that someone is distilling your model
Distillation leaves fingerprints in traffic, accounts, and prompts. Look for synchronized activity across many accounts, shared payment details, prompts that ask for hidden reasoning, and sudden shifts toward your newest model. None of these signals is proof on its own. Several together deserve an investigation.
| Signal | What it looks like | Seen in |
|---|---|---|
| Synchronized traffic | Many accounts with identical patterns and timing | Anthropic, DeepSeek campaign |
| Shared payment or infrastructure | Accounts tied by payment methods, IP ranges, or request metadata | Anthropic attribution methods |
| Reasoning fishing | Requests to output prior reasoning verbatim, translate it, or tag it | Anthropic, October 2026 |
| Grading tasks at volume | The model used as a reward model for another model's training | Anthropic, DeepSeek campaign |
| Fast pivots | Traffic shifts to a new release within a day | Anthropic, MiniMax campaign |
| Language-forcing instructions | Prompts that force reasoning into a target language | Google, via CSO Online |
Ordinary users rarely ask a model to write out its full hidden reasoning. That is why reasoning-fishing requests make a strong detection signal. Anthropic built its classifiers around that observation.
Watch the commercial layer too. Resellers and proxy services can make a hydra cluster look like one large, legitimate customer. Contract terms that require resellers to pass end-user identity, and to act on abuse reports, close part of that gap.
Defenses that work, and what they cost
Effective defenses protect reasoning, block extraction attempts, and make repeated attempts expensive. According to Anthropic (October 2026), early evidence suggests its new methods have reduced the data harvested by distillers by around 90% to 99%. Every control has trade-offs, including integration changes and a small rate of blocked legitimate requests.
| Defense | What it stops | Trade-off |
|---|---|---|
| Summarize or withhold reasoning | Passive harvesting of reasoning traces | Less transparency for legitimate users |
| Bind reasoning to the requesting account | Replay of reasoning across accounts and models | Integrations may need updates |
| Extraction classifiers | Active attempts to reveal hidden reasoning | Rare false positives |
| Stricter limits and fast bans | Hydra clusters and persistent accounts | Needs strong account signals |
| Rate limits and quotas | Bulk querying from single sources | Can slow heavy legitimate users |
| Watermarking outputs | Helps prove unauthorized reuse later | Detection, not prevention |
| Limit output detail | Parameter extraction through logits | Fewer options for advanced users |
According to Anthropic (2026), its two-stage classifier now catches all the distillation techniques described in its earlier reports, with a false positive rate under 0.1%. It bans associated accounts, often within two hours of a campaign starting. These are vendor-reported results on Anthropic's own platform.
The trade-offs are honest. Anthropic said preserved thinking forced some developers to update integrations, and it now charges for requests its safeguards block. You will face similar choices.
The research side adds a warning about output detail. Carlini et al. (2024) showed that detailed API outputs let them recover a production model's projection matrix for under $20. The same team estimated under $2,000 for gpt-3.5-turbo's full matrix. If your API exposes probabilities, ask whether customers really need them.
According to OWASP (2025), unbounded consumption includes model extraction and functional model replication. Its mitigations include rate limiting, user quotas, and watermarking frameworks.
Security and governance: protecting agents you expose
Distillation sits alongside the classic agent risks, and the controls overlap. IBM (2026) found that 92% of organizations with an AI-related breach lacked proper AI access controls. Breaches involving model inversion, which extracts sensitive data from a model, averaged $6.07 million. Identity and access are the shared foundation.
Three risks share that foundation:
- Prompt injection: the same crafted prompts that fish for reasoning can also pull system prompts, internal data, or tool outputs your agent should never reveal.
- Data leakage: a distiller collecting millions of responses may also collect sensitive records an agent returns by mistake.
- Tool misuse: fraudulent accounts with access to agent tools can run up costs or trigger actions at scale.
NeuralTrust approaches this with one clear layer in front of the models and agents you expose. You control who reaches them through the Agent Gateway (TrustGate). You protect prompts, responses, and tool calls with Agent Runtime Security (TrustGuard). AI Red Teaming (TrustTest) lets you test system-prompt extraction and prompt injection attempts before attackers do.
For more on the risk side, our guide to AI model theft covers weight theft and infrastructure controls. Our enterprise guide to AI agent security covers the wider agent threat model.
Which should you choose?
Your exposure depends on what you offer and to whom. A company selling a fine-tuned model through a public API carries the highest risk. An internal-only agent carries the least. The table below matches common situations to the controls worth starting with.
| If you... | Start with | Why |
|---|---|---|
| Sell access to a fine-tuned model or agent | Account verification, extraction detection, and quotas | Your outputs are the product being copied |
| Expose agents to partners or resellers | Contract terms on end-user identity, plus per-partner limits | Proxy networks hide behind resellers |
| Run agents only for employees | Identity, logging, and output filtering | Insiders and compromised accounts remain the path |
| Show reasoning or plans to users | Summaries instead of full traces | Full traces are the most valuable training data |
| Expose token probabilities | Remove or coarsen them unless required | Detailed outputs enable parameter extraction |
| Plan to adopt a cheaper third-party model | Red teaming before production | Distilled models may lack matching safeguards |
Conclusion
Distillation turns your model's strengths into someone else's shortcut. The 2026 cases show it runs at industrial scale, targets agentic skills, and adapts within a day. For AI agent security teams, the response starts with identity and access, then adds extraction detection, reasoning protection, and quotas. Vendor results suggest those layers can cut harvested data sharply. Test them on your own agents before a fraudulent account does it for you.
Secure Exposed AI Agents in Production with NeuralTrust
Control who reaches your models and agents, and keep tabs on every prompt and response.
)
Related Comparisons
- AI Model Theft: How to Prevent It
- AI Agent Security for Enterprises: Complete Guide
- Why Your AI Model Might Be Leaking Sensitive Data
- Rate Limiting & Throttling for AI Agents
FAQs about AI Agent Security
1. What is a distillation attack in AI?
A distillation attack is the unauthorized use of a model's outputs to train a cheaper copy. Attackers query the target at scale, often through many fraudulent accounts, and train a student model on the responses. Reasoning traces are the prize, because they show how the model reaches answers. Google calls it a form of intellectual property theft; Anthropic says it violates its terms of service.
2. What is model extraction?
Model extraction is the broader family of attacks that recover a model's behavior or internals through queries. It includes distillation, which copies capabilities, and parameter extraction, which recovers parts of the weights. Carlini and colleagues showed in 2024 that detailed API outputs could reveal a production model's projection matrix for under $20 in queries.
3. How do you prevent distillation attacks?
Combine account verification, rate limits, and detection of extraction patterns such as requests for hidden reasoning. Summarize or withhold reasoning traces, bind them to the requesting account, and ban persistent offenders quickly. Watermarking can help prove reuse later. Anthropic reported that after it introduced its classifier, distillers extracted 10 to 100 times fewer reasoning tokens on its platform.
4. Is AI model distillation legal?
Distillation itself is a legitimate technique, and some cloud providers offer it as a service. Distilling another company's model without permission usually breaches its terms of service, which is how Google and Anthropic describe the 2026 campaigns. Whether it also breaks intellectual property or trade secret law depends on the jurisdiction. Ask your legal team for advice.
5. What are the main AI agent security risks?
The most common risks are prompt injection, data leakage, tool misuse, and identity or privilege abuse. OWASP's agentic list covers these in detail. Distillation adds an outbound risk: someone harvesting your agent's outputs to copy its capabilities. IBM's 2026 report found 92% of organizations with AI-related breaches lacked proper AI access controls.
6. How do you secure AI agents?
Give each agent its own identity and least-privilege access. Route model and tool traffic through one enforcement point that inspects prompts, responses, and tool calls. Log every decision, red team agents before launch and after changes, and require human approval for irreversible actions. For agents you expose externally, add quotas and extraction detection.
7. Can distillation attacks be detected?
Yes, often. Distillation traffic differs from normal use: synchronized accounts, shared payment details, high volumes, and prompts that ask for hidden reasoning. Anthropic says it attributed campaigns using IP correlation, request metadata, and infrastructure indicators. Detection works best when traffic analytics, account signals, and prompt classifiers feed the same investigation.
About the Author
Roger Howroyd is Head of Global SEO and AI at NeuralTrust, where he leads the company's search strategy across SEO, AEO, GEO, and LLM optimization. He specializes in AI-powered search, content strategy, and SEM. Connect on LinkedIn.
NeuralTrust is the leading platform for securing and scaling AI agents. Named a Pioneer in the Gartner Emerging Market Quadrant for AI Application Security 2026, recognized across four Gartner Hype Cycle reports in 2026, and featured in the Gartner Market Guide for Guardian Agents 2026, the Gartner Market Guide for AI Gateways 2025 and the KuppingerCole Leadership Compass for Generative AI Defense 2025. Headquartered in Barcelona with offices in London and New York. ISO 27001 certified.
Sources
- Anthropic, "Detecting and preventing distillation attacks," February 23, 2026. https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks
- Anthropic, "Strengthening our defenses against distillation attacks," October 12, 2026. https://www.anthropic.com/news/distillation-defenses
- CSO Online, "Google fears massive attempt to clone Gemini AI through model extraction," February 13, 2026. https://www.csoonline.com/article/4132098/google-fears-massive-attempt-to-clone-gemini-ai-through-model-extraction.html
- Carlini et al., "Stealing Part of a Production Language Model," arXiv, March 11, 2024. https://arxiv.org/abs/2403.06634
- OWASP GenAI Security Project, "LLM10:2025 Unbounded Consumption," 2025. https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/
- IBM and Ponemon Institute, "Cost of a Data Breach Report 2026," July 29, 2026. https://assets.ibm.com/adobe/assets/urn:aaid:aem:21111142-1251-4369-86fb-57b82f5bb108/original/as/Cost%20of%20a%20Data%20Breach%20Report%202026.pdf
- Gartner, "Gartner Says Aligning AI Value, Risk and Safety Is Now the Top Challenge for CIOs," November 9, 2026. https://www.gartner.com/en/newsroom/press-releases/2026-11-09-gartner-says-aligning-artificial-intelligence-value-risk-and-safety-is-now-the-top-challenge-for-chief-information-officers
- Google Threat Intelligence Group, "GTIG AI Threat Tracker: Distillation, Experimentation, and (Continued) Integration of AI for Adversarial Use," February 12, 2026. https://cloud.google.com/blog/topics/threat-intelligence/distillation-experimentation-integration-ai-adversarial-use
)
)
)
)