)
What is LLM batch inference?
LLM batch inference is the practice of grouping multiple AI requests into a single job and processing them asynchronously, rather than sending each one in real time. OpenAI's Batch API and Anthropic's Message Batches API both offer 50% cost discounts for async batch processing, with results delivered within 24 hours. The trade-off is latency: batch is not for live user interactions, but for any workload where a human is not waiting for the answer right now.
TL;DR - Key Takeaways
- The OpenAI Batch API and Anthropic Message Batches API both offer 50% cost discounts versus synchronous APIs for async processing
- OpenAI supports up to 50,000 requests per batch file (200 MB max): Anthropic supports up to 10,000 requests per batch
- Both providers guarantee 24-hour completion: Anthropic batches typically finish in under 1 hour
- Batch rate limits are a separate pool from your standard limits, using them does not consume your synchronous quota
- Batching suits classification, document processing, model evaluation, dataset labeling, and embedding generation
- Batching does not suit real-time chat, live safety gates, or any flow where a human is waiting for the answer
- Anthropic's batch discount stacks with prompt caching, compounding savings further
- NeuralTrust TrustGate applies batch routing policies automatically at the gateway layer
Half price. Same model. Same output quality. Just not instant. If you have workloads that do not need a real-time answer, you are leaving 50% of your AI budget on the table. This guide walks through when to batch, how to implement it on OpenAI and Anthropic, and what breaks if you do it wrong.
)
Why Are You Paying Real-Time Prices for Work That Does Not Need a Real-Time Answer?
Picture your Monday morning pipeline. Your system runs 40,000 customer support tickets through a classifier to tag them by category and priority. Nobody is sitting there waiting. The results feed into a report that your ops team reads at noon.
You are sending those 40,000 requests synchronously. One by one. Paying full price for each one.
That is the problem. Most teams default to synchronous inference because it is simpler to set up. You call the API, you get the answer, you move on. But synchronous inference is priced for speed. You are paying a premium for sub-second latency on workloads that do not need it.
OpenAI and Anthropic both noticed this. Both built a solution. Both price it the same: 50% off for workloads you are willing to run asynchronously.
That is not a small discount. At scale, it is the difference between an AI budget that is sustainable and one that is not.
)
How Batching Works
The mechanic is simple. Instead of sending 1,000 requests one at a time, you bundle them into a file and submit them as a single job. The provider processes the job asynchronously and gives you the results when it is done.
No magic. No retraining. Same models, same prompts, same output quality. Just a different delivery mechanism.
Three things change:
- Timing. You do not get the answer in 300ms. You get it within 24 hours. In practice, much faster: Anthropic reports most batches completing in under 1 hour.
- Cost. 50% discount on both input and output tokens compared to synchronous API calls. Both OpenAI and Anthropic apply this identically.
- Rate limits. Your batch requests come out of a separate quota. OpenAI's Batch API rate limits are a separate pool, they do not touch your standard per-model limits. Running a large batch job will not throttle your production real-time traffic.
OpenAI Batch API
The OpenAI Batch API accepts a JSONL file where each line is a self-contained API request. Submit the file, kick off the job, poll for status, retrieve results.
Key specs:
- Up to 50,000 requests per batch file
- File size limit: 200 MB
- Completion window: 24 hours
- Discount: 50% off input and output tokens vs synchronous API
- Supported models: GPT-4o, GPT-4o mini, o3-mini, and others
Each line in your JSONL looks like a standard chat completion request, wrapped in a custom_id field you use to match results back to inputs. Error handling is your responsibility: if individual requests fail within the batch, the job still completes, you check per-request status in the output file.
The separate rate limit pool is a big deal in practice. Teams running high-traffic synchronous workloads have historically been blocked from batching because they feared consuming their quota. With a separate pool, that concern is gone.
Anthropic Message Batches API
The Anthropic Message Batches API follows the same pattern with a few differences in the limits.
Key specs:
- Up to 10,000 requests per batch
- Completion: typically under 1 hour, guaranteed within 24 hours
- Discount: 50% off input and output tokens
- Supported models: Claude Sonnet 4.6, Claude Haiku 4.5, and others
- Available on Anthropic API, AWS Bedrock, and Google Cloud
One compounding advantage: Anthropic's batch discount stacks with prompt caching discounts. If your batch requests share a common system prompt or large context block, the cached token discount applies on top of the 50% batch discount. That combination can push effective per-token costs significantly lower than either discount alone.
The implementation is straightforward. You submit a list of requests objects, each with a custom_id and a standard params block matching the Messages API format. Poll the batch status via a simple GET request, retrieve results when it shows as ended.
Which Workloads Are Right for Batching
This is the decision that matters. Batch is not for everything. Get this wrong and you will frustrate users or break safety-critical flows.
Batch fits well:
-
Classification and tagging at scale. Categorizing tickets, labeling content, routing documents. Nobody waits for these in real time.
-
Document processing. Summarization, extraction, and transformation jobs that run on a fixed dataset. The results feed a downstream system, not a live user.
-
Model evaluation and benchmarking. Running thousands of eval prompts against a new model version or prompt change. You need the results before a deployment decision, not before a user interaction.
-
Dataset labeling and augmentation. Using an LLM to annotate or clean training data. Entirely offline. Batch is ideal.
-
Embedding generation. Building or refreshing a vector index. High volume, no latency requirement.
-
Nightly and weekly reports. Any scheduled analytics job that aggregates LLM outputs into a dashboard or report. Batch does not fit:
-
Real-time chat and copilots. Users are waiting. A 24-hour response is not a response.
-
Live safety gates. If the LLM output determines whether a user's action is allowed, that decision needs to be synchronous.
-
Fraud and anomaly detection. Any flow that must block or flag before a transaction or event completes.
-
Streaming UX. If the experience depends on seeing tokens arrive incrementally, batch breaks the UX entirely.
)
The Latency-Cost Trade-Off
The 50% discount is real. So is the latency. You need to understand both before you commit a workload to batch.
Real-time inference: answers in 300-800ms. Costs 2x more per token. Right for any user-facing interaction.
Async batch inference: answers in minutes to hours. Costs half as much. Right for any workload you can afford to run in the background.
The mistake teams make is treating this as binary. It is not. Most production systems should run both in parallel: synchronous for live user traffic, batch for the offline processing layer that runs alongside it.
Your nightly re-ranking job? Batch. Your support ticket classifier that runs every 15 minutes on accumulated tickets? Batch. Your live chat assistant? Synchronous. Your A/B test evaluation that runs on yesterday's conversations? Batch.
The key question for every LLM call in your system: is a human waiting for this answer right now? If no, you have a batch candidate.
Batch Comparison at a Glance
| OpenAI Batch API | Anthropic Message Batches | |
|---|---|---|
| Discount | 50% off input + output | 50% off input + output |
| Max requests per batch | 50,000 | 10,000 |
| Max file size | 200 MB | N/A |
| Completion SLA | 24 hours | 24 hours |
| Typical completion | Hours | Under 1 hour |
| Rate limit pool | Separate from sync | Separate from sync |
| Stacks with prompt cache | Yes | Yes |
| Error handling | Per-request in output file | Per-request in results |
What Breaks and How to Handle It
Batch processing shifts responsibility for error recovery to you. Unlike synchronous calls, where a failed request is an exception you catch immediately, a batch job completes even if some individual requests inside it failed.
Three things to build into your implementation:
1. Per-request status checks.
When you retrieve batch results, check each request's status field individually. A top-level "succeeded" batch may still have failed individual items. Build your parsing logic to separate successes from failures before you process results.
2. Retry logic for failed items.
Extract failed request IDs, reconstruct the original inputs, and resubmit as a new batch or via synchronous fallback. Do not assume a complete batch means complete results.
3. Idempotent downstream processing.
Your system that consumes batch results should be able to process the same output file twice without producing duplicate effects. Batches occasionally re-deliver results or experience partial retrieval issues. Make your consumer idempotent.
The AI token usage monitoring guide covers how to instrument batch jobs alongside your synchronous traffic so your cost and attribution data stays consistent regardless of which path a request takes.
Enforcement at the Gateway Layer
Implementing batch routing at the application level works. The problem is consistency: every team needs to know which workloads to batch, every new pipeline starts with synchronous defaults, and there is no enforcement if a team forgets.
NeuralTrust TrustGate applies batch routing policies at the gateway layer. Routes you configure as batch-eligible are automatically directed to the batch API, with max_tokens, model, and attribution tags applied consistently. Your synchronous pool stays clean. Your batch pool is used where it should be.
This pairs with the LLM Model Routing guide: route by complexity at the model level, and by latency tolerance at the sync/batch level. Both decisions happen at the gateway, not in application code.
For the full picture on cost reduction levers, the LLM Cost Reduction guide and the AI Token Optimization pillar cover how batching fits alongside caching, compression, and routing in a complete cost strategy.
Frequently Asked Questions about LLM Batching and Async Inference
1. When should I use batch inference instead of real-time inference?
Use batch inference when no human is waiting for the result in real time. Good candidates: classification pipelines, document processing, model evaluation, dataset labeling, embedding generation, and any scheduled reporting job. Stick with synchronous inference for live chat, safety gates, fraud detection, and any UX where the user sees the response as it generates.
2. How much does the OpenAI Batch API cost?
The OpenAI Batch API costs 50% less than the standard synchronous API for both input and output tokens. The discount applies to all supported models. Batch API rate limits are a separate pool from your standard limits, so batch jobs do not affect your synchronous quota.
3. How long do batch jobs take to complete?
OpenAI guarantees completion within 24 hours. Anthropic guarantees 24 hours but reports most batches completing in under 1 hour. Actual completion time depends on queue depth at the time of submission and the size of your batch.
4. Can I use batch processing with prompt caching?
Yes. On Anthropic's API, the Message Batches discount stacks with prompt caching discounts. If your batch requests share a large common context block (system prompt, documents), the cached input token discount applies on top of the 50% batch discount, compounding your savings.
5. What happens if some requests in my batch fail?
The batch job completes even if individual requests inside it fail. You are responsible for checking per-request status in the output file, extracting failed items, and resubmitting them. Build your result parser to separate successes from failures before downstream processing.
Related articles:
- AI Token Optimization: Complete Guide to Reducing LLM Costs
- Prompt Compression: Cut Token Costs Without Losing Quality
- LLM Caching Strategies: Prompt Caching, Semantic Caching, and When to Use Each
- LLM Cost Reduction: 12 Strategies to Cut AI Inference Costs
- Context Window Optimization: 6 LLM Strategies for 2026
- LLM Model Routing: Route Queries to the Right Model Automatically
- Token Usage Monitoring: Track, Attribute, and Optimise AI Spend
- Output Length Control: Stop LLMs Over-Generating & Wasting Tokens
About the Author
Roger Howroyd is Head of Global SEO and AI at NeuralTrust, where he leads the company's search strategy across SEO, AEO, GEO, and LLM optimization. He specializes in AI-powered search, content strategy, backlink development, and SEM. Connect on LinkedIn.
NeuralTrust is an AI agent security platform, recognized in the Gartner 2025 Market Guide for AI Gateways and Guardian Agents, the Gartner Hype Cycle for Application Security 2026, and the KuppingerCole 2025 Leadership Compass for Generative AI Defense. ISO 27001 certified. Headquartered in Barcelona.
)
)
)