)
Prompt compression is the practice of removing low-information tokens from LLM inputs before sending them to a model, reducing token count by 20 to 80% while preserving the semantic content that determines output quality.
The compressed prompt costs less to process, returns faster, and often produces the same or better output than the original. A 40% shorter prompt is not a compromise. It is often a better prompt.
TL;DR - Key Takeaways
- Prompt compression removes redundant, low-information tokens from LLM inputs before inference. The target model sees less text but loses almost none of the meaning.
- LLMLingua (Microsoft Research) achieves up to 20x compression with under 2% quality loss on benchmarks including CoQA, HotpotQA, and TriviaQA. LongLLMLingua achieves a 17.1% performance improvement at 4x compression on long-context tasks by filtering noise.
- Selective Context, the technique that established the field, achieved 50% context reduction while maintaining comparable performance: only 0.023 degradation in BERTScore.
- RAG pipelines are the highest-yield target: retrievals are reliably redundant, compression ratios are highest, and quality impact is lowest because you are removing context the model was not using.
- Manual compression (restructuring verbose system prompts, replacing prose instructions with structured lists, cutting redundant examples) requires no tooling and often reduces tokens by 20-40% with zero quality loss.
- NeuralTrust TrustGate enforces prompt size policies at the gateway layer, so compression rules apply consistently across every application, not just the ones where engineering remembers to implement them.
What is prompt compression?
Last quarter, an engineering team I know audited their flagship RAG application. The system prompt had grown from 800 tokens at launch to 4,200 tokens eighteen months later. Nobody had deliberately added 3,400 tokens. They had accumulated in the usual way: a few edge-case instructions here, a clarifying example there, a paragraph explaining a policy that had since changed but nobody deleted.
Those 3,400 extra tokens went into every single request. At $3 per million input tokens on Sonnet 4.6, running 50,000 requests per day, that is $153 per day in system prompt waste alone. Not model capability. Not useful context. Just waste.
Prompt compression is the practice of removing low-information tokens from LLM inputs before sending them to a model through manual restructuring, extractive techniques, or automated compression models, to reduce cost and latency without degrading output quality.
The underlying insight, established in the Selective Context paper (Li et al., 2023), is that not all tokens contribute equally to model comprehension. A large language model's understanding of a prompt is shaped primarily by the high-information tokens, the ones that carry distinctive meaning. The filler tokens (conjunctions, hedge phrases, repeated context, redundant examples) can often be removed with minimal effect on what the model produces.
This is not lossy compression in the image sense. You are not approximating the prompt. You are removing the parts that were not doing work in the first place.
)
Why system prompts accumulate waste over time?
System prompts are not written once and left alone. They grow. Every edge case someone encounters, every misunderstanding a user reported, every compliance requirement added after launch, all of it gets appended. Rarely does anything get removed. The result, after twelve to eighteen months in production, is a prompt that is three to five times larger than it needs to be.
The patterns are consistent across teams:
1. Redundant instructions.
The same constraint expressed in three different phrasings because different people added it at different times. "Do not include personal contact information" appears in the opening paragraph, again in the format instructions, and again in the closing policy block.
2. Verbose examples.
A five-sentence worked example showing the correct output format when a one-sentence example would demonstrate the same pattern. Examples are among the most token-expensive elements of a system prompt.
3. Stale context.
Instructions for handling a feature that was deprecated. Explanations of a policy that changed. Edge-case handling for a user flow that no longer exists.
4. Prose where structure would do.
A paragraph explaining five formatting rules when a five-item list would communicate the same rules in half the tokens.
None of this is unique to large teams or complex products. It is the default entropy of any prompt that is actively used and maintained by more than one person.
Manual compression techniques: no tooling required
This is where to start. Manual compression requires no infrastructure, no new dependencies, and no latency overhead. Done carefully, it typically reduces system prompt tokens by 20 to 40% without any quality regression.
Technique 1: Audit for duplicate instructions
Read through the full system prompt looking specifically for the same constraint or rule expressed more than once. Consolidate to the clearest single instance. This alone typically finds 5 to 15% of system prompt tokens to eliminate.
Technique 2: Replace prose with structured lists
A paragraph that explains five rules takes more tokens than a numbered list of five rules, and the list is easier for the model to follow. Convert any instructional prose to bullet or numbered format.
Technique 3: Shorten examples to the minimum that demonstrates the pattern
If your system prompt includes worked examples to show output format, ask: what is the shortest example that still demonstrates the pattern? A one-turn example with a short input and short output demonstrates format just as well as a five-sentence example in most cases. If you need multiple examples, prefer one good one over three mediocre ones.
Technique 4: Remove stale instructions
Review every instruction in the system prompt and ask: is this still relevant to how the system is actually used today? Stale instructions add tokens and can actively confuse the model if they conflict with current behavior.
Technique 5: Use token-efficient delimiters
### instead of ======= SECTION SEPARATOR =======. Q: and A: instead of User question: and Model answer:. These small substitutions add up over a long prompt.
)
Automated compression: LLMLingua, Selective Context, and when to use each
Manual compression handles accumulated waste in system prompts. Automated compression handles dynamic content (retrieved documents, long conversation histories, injected context) where you cannot manually review every input at runtime.
1. Selective Context
Selective Context (Li et al., 2023, arxiv:2310.06201) established the research foundation for the field. It calculates the self-information of each lexical unit (phrases, sentences, or tokens) using a small language model, and filters out units whose information content falls below a threshold.
Benchmarks show 50% context reduction while maintaining comparable performance, with only 0.023 degradation in BERTScore and 0.038 degradation in faithfulness metrics. The technique works particularly well on retrieved context and conversation history: long inputs with genuinely redundant content.
Best for: Conversation history compression, retrieved document pruning, any input where the content is variable and often redundant.
2. LLMLingua
LLMLingua (Jiang et al., Microsoft Research, 2023) extended this approach with a coarse-to-fine compression framework. It uses a small language model to calculate token perplexity: tokens with lower perplexity contribute less information entropy and can be safely removed.
A budget controller allocates compression ratios across different segments of the prompt, preserving instruction-critical sections at higher fidelity while compressing examples and context more aggressively.
Results from the original paper: up to 20x compression with under 2% performance loss on CoQA, HotpotQA, and TriviaQA benchmarks. At 5x compression on a workload spending $20,000 per month on input tokens, that is $16,000 saved monthly with no model changes.
3. LLMLingua-2
LLMLingua-2 (Pan et al., Microsoft Research, ACL 2024) reformulates compression as a token classification problem, training a BERT-level encoder via data distillation from GPT-4. It outperforms LLMLingua across all benchmarks at equivalent compression ratios, with faster inference due to a smaller compressor model. It reduces end-to-end latency by up to 2.9x at 2x to 5x compression ratios.
4. LongLLMLingua
LongLLMLingua (Jiang et al., 2024) is optimized for long-context scenarios: the exact use case that RAG systems create. It uses question-aware compression that evaluates the relevance between context segments and the query using contrastive perplexity.
Results: up to 21.4% performance improvement at approximately 4x fewer tokens on NaturalQuestions. Better outputs, fewer tokens. That is not a tradeoff. That is a win in both directions.
| Technique | Compression ratio | Quality impact | Best use case |
|---|---|---|---|
| Manual restructuring | 20-40% reduction | None if done carefully | System prompts, static instructions |
| Selective Context | ~50% reduction | 0.023 BERTScore degradation | Conversation history, retrieved docs |
| LLMLingua | Up to 20x | Under 2% on reasoning tasks | General-purpose dynamic context |
| LLMLingua-2 | 2-5x typical | Lower than LLMLingua, faster | Task-agnostic production workloads |
| LongLLMLingua | ~4x | +17.1% on long-context tasks | RAG pipelines, long retrieved context |
)
Quality-vs-compression tradeoffs: what the research actually says
There is a common assumption that compression always trades quality for cost. The research does not support this for most production use cases.
A 2024 study comparing compression methods on multi-document question answering found that extractive reranker-based compression achieved +7.89 F1 points on 2WikiMultihopQA at 4.5x compression. Compression improved accuracy by filtering noise. The same study showed abstractive compression at similar ratios decreased performance by 4.69 F1 points. Extractive compression often outperforms abstractive approaches: you are removing tokens that the model was not using well anyway.
The practical implication: your starting point matters. A verbose system prompt with redundant instructions, a RAG context with 20 retrieved chunks when 5 are relevant, a conversation history that faithfully preserves every "thanks, got it", these are cases where compression improves both cost and quality simultaneously. The model gets a cleaner signal. You pay less for it.
Where compression genuinely trades quality for cost: highly structured reasoning chains where intermediate steps matter, precise technical specifications where specific phrasing is required by downstream systems, and code that cannot be paraphrased without breaking functionality. These are the cases to exclude from automated compression.
Before/after examples with token counts
Example 1: System prompt instruction block
Before (94 tokens):
After (41 tokens):
Token reduction: 56%. Semantic content preserved: 100%.
Example 2: RAG context injection
Before: 12 retrieved document chunks, 3,200 tokens. Many chunks contain general background information rather than query-specific content.
After LongLLMLingua (4x compression): 800 tokens. Query-relevant content is preserved; general background filtered. Result on NaturalQuestions-style tasks: performance improvement of up to 21.4%, not degradation.
FAQs about prompt compression
1. What is prompt compression?
Prompt compression is the removal of low-information tokens from LLM inputs before sending them to a model. A compression process (either manual restructuring or an automated model like LLMLingua) identifies tokens that contribute less information entropy and removes them, producing a shorter prompt that preserves the semantic content the model needs to generate an equivalent response. The target model sees less text, processes it faster, and the result is billed at the lower token count.
2. How much can prompt compression reduce token costs?
It depends on your prompt structure and task type. Manual restructuring of verbose system prompts typically achieves 20-40% reduction with no quality loss. Selective Context achieves approximately 50% reduction with 0.023 BERTScore degradation. LLMLingua achieves up to 20x compression with under 2% performance loss on benchmarks. LongLLMLingua achieves 4x compression with a 17.1% performance improvement on long-context tasks. RAG pipelines are the highest-yield target because retrieved context is reliably redundant.
3. Does prompt compression degrade output quality?
Not always, and sometimes it improves quality. A 2024 study on multi-document QA found extractive compression improved F1 scores by 7.89 points at 4.5x compression by filtering noise the model was not using effectively. Quality does degrade for tasks that require precise technical specifications, complete reasoning chains, or exact code structures. These should be excluded from automated compression. For the typical system prompt or RAG context, compression removes tokens the model was not benefiting from.
4. What is the difference between LLMLingua and LLMLingua-2?
LLMLingua uses a small language model to calculate token perplexity and removes low-perplexity tokens, achieving up to 20x compression with under 2% quality loss. LLMLingua-2 reformulates compression as a token classification problem trained via data distillation from GPT-4, using a BERT-level encoder. It outperforms LLMLingua across all benchmarks at equivalent compression ratios and is faster due to the smaller compressor model. For production workloads, LLMLingua-2 is the more practical choice.
5. Where should I start with prompt compression?
Start with manual compression of your system prompts. No tooling required, no latency overhead, and 20-40% token reduction is realistic. Audit for duplicate instructions, convert prose to structured lists, shorten examples, and remove stale instructions. Once you have cleaned up the static elements, apply automated compression to dynamic inputs: retrieved context in RAG systems is the highest-yield target.
Key Takeaways
- Prompt compression removes tokens that are not doing meaningful work. The model produces equivalent or better output at a fraction of the input cost.
- Start manually. Most system prompts have 20-40% of tokens that can be eliminated without any tooling: duplicate instructions, verbose examples, stale content, prose where structured lists would serve.
- For dynamic content (RAG retrieved context, conversation history, long injected context) LLMLingua-2 and LongLLMLingua are production-ready tools with documented benchmark results from Microsoft Research.
- Compression often improves quality, not just cost. Filtering noise from RAG context is a documented way to boost accuracy, not just reduce spend.
- NeuralTrust TrustGate enforces prompt size policies at the gateway layer across all connected applications, making compression systematic rather than application-by-application.
- For the complete token optimization strategy that compression fits into, see our AI Token Optimization Complete Guide.
Related Articles
About the Author
Roger Howroyd is Head of Global SEO and AI at NeuralTrust, where he leads the company's search strategy across SEO, AEO, GEO, and LLM optimization. He specializes in AI-powered search, content strategy, backlink development, and SEM. Connect on LinkedIn
NeuralTrust is an AI agent security platform, recognized in the Gartner 2025 Market Guide for AI Gateways and Guardian Agents, and the KuppingerCole 2025 Leadership Compass for Generative AI Defense. Headquartered in Barcelona with ISO 27001 certification.
)
)
)
)