Design efficient batch processing strategies

The Message Batches API lets you submit a large collection of independent Message requests in a single asynchronous job instead of making one blocking API call at a time. You POST a batch to /v1/messages/batches, the platform processes the requests in the background, and you poll the batch's status until it finishes, then download the results file.

The economics and the constraints are the whole point. Batched tokens cost about 50% less than the equivalent synchronous request for both input and output. In exchange, the batch processes asynchronously with a window of up to 24 hours. Most batches finish well before that, but there is no guaranteed latency SLA, so you must design as if any batch could take the full 24 hours. Batches have large capacity (on the order of 100,000 requests or a few hundred MB per batch), and results remain retrievable for weeks after completion.

POST /v1/messages/batches
{
  "requests": [
    { "custom_id": "invoice-001",
      "params": { "model": "claude-...", "max_tokens": 1024,
                  "messages": [ { "role": "user", "content": "..." } ] } },
    { "custom_id": "invoice-002", "params": { "...": "..." } }
  ]
}

Each entry in requests[] is a completely standalone Message call with its own params. The batch is just a container that runs them cheaply and asynchronously; it does not chain them together or share state between them.

The core exam decision is choosing batch versus the synchronous (real-time) /v1/messages API based on whether a human or a downstream step is blocked waiting for the answer. Batch is appropriate for non-blocking, latency-tolerant workloads: overnight technical-debt reports, weekly compliance audits, nightly test generation, and bulk document extraction that feeds a report someone reads the next morning. Nothing is stalled while the batch runs, so a multi-hour turnaround is free money.

Batch is inappropriate for blocking workflows. The canonical disqualifier is a pre-merge check that a developer waits on before they can merge a pull request. If someone is sitting there waiting, or a synchronous pipeline stage cannot proceed until the answer arrives, the up-to-24-hour window makes batch unusable regardless of the cost savings. The right move is to split the workloads: keep the synchronous API for the blocking pre-merge check, and move only the latency-tolerant overnight report to batch.

A classic distractor is "switch both to batch and poll for completion" or "switch both with a real-time fallback if the batch is slow." Both are wrong. Batch often finishes fast, but "often" is not acceptable for a blocking gate, and bolting a real-time fallback onto a batch job just adds complexity to hide the fact that the workload never belonged in batch. Match each workflow to the API that fits its latency requirement.

Every request in a batch carries a custom_id that you assign. It is the key that ties each result back to the input it came from. This matters because batch results are not guaranteed to come back in the order you submitted them, and each request succeeds or fails independently. You must correlate by custom_id, never by array position or submission order.

Each result reports its own outcome, typically one of succeeded, errored, canceled, or expired. That per-request status is what lets you treat a batch as a set of independent jobs rather than an all-or-nothing unit. A common exam misconception is that batch results are unusable because ordering is lost; the correct rebuttal is that custom_id fully solves correlation, so ordering is a non-issue.

In practice you keep a map from custom_id to your domain object (an invoice number, a file path, a PR file, a ticket id). When results come back, you look up each one by its custom_id, write the succeeded outputs to your store, and collect the custom_ids of the errored ones for targeted resubmission. Make custom_id values stable and meaningful (for example the document id) so a resubmission batch reuses the same identifiers and stays traceable end to end.

A single batch request is one independent completion, not an agentic loop. The batch API does not support multi-turn tool calling within a request: Claude cannot call a tool, have your code execute it, feed the result back, and continue reasoning, all inside one batched request. There is no mechanism for your infrastructure to run a tool and return its output mid-request during batch processing.

You can still include tool definitions in a batch request to get a single, schema-validated tool_use output (this is a common structured-extraction pattern, and it pairs naturally with domain 4.3). What you cannot do is orchestrate the full agentic loop, that is send request, inspect stop_reason, execute the tool, append the result, and iterate, inside the batch itself.

The design implication is a clean split. If a task needs Claude to actually invoke tools and act on their results (browsing, querying a database, running code), that agentic loop belongs in the synchronous API or the Agent SDK, where your loop can execute tools between turns. Batch is for large volumes of self-contained, single-shot completions: classify this document, extract these fields, summarize this file. If you find yourself wanting tool execution mid-request in a batch, that is a signal the workload is agentic and should not be batched as one request.

When latency-tolerant work still has a deadline, you size how often you submit batches so the worst case fits the SLA. The reasoning: a document that arrives just after a submission waits until the next window (up to the full window length W), then the batch itself can take up to 24 hours. So worst-case latency is approximately W + 24 hours.

To guarantee a 30-hour SLA with a 24-hour batch window, you need W + 24 <= 30, so W <= 6 hours. Choosing a 4-hour submission window satisfies this with margin: worst case 4 + 24 = 28 hours, leaving a 2-hour safety buffer for retrieval, downstream processing, and jitter. This is exactly the blueprint's example: 4-hour windows guarantee a 30-hour SLA under 24-hour batch processing.

worst_case_latency = submission_window + max_batch_processing
28h                = 4h                 + 24h        <= 30h SLA  (OK, 2h buffer)

Note what this budget does not include: a full resubmission cycle. If a batch's failures need to be resubmitted, that is another window plus up to 24 hours, which would blow a tight SLA. That is why maximizing first-pass success (next concept) is part of hitting an SLA, not a separate nicety. Shorter windows reduce worst-case latency but submit more, smaller batches; longer windows batch more efficiently but eat into the SLA.

Because a failed batch item costs you another window plus processing time, the cheapest strategy is to get as many requests right on the first pass as possible. Before batching a large volume, refine the prompt on a small representative sample using the synchronous API where iteration is fast. Fix the format, schema, and edge-case handling on the sample, then run the full batch once the sample passes reliably. This trades a little up-front sync cost for far fewer expensive resubmission rounds.

When a batch does come back with failures, resubmit only the failed items, identified by custom_id, not the whole batch. Reprocessing the succeeded requests would waste the discount you just captured. Apply targeted modifications to the failures based on why they failed: chunk documents that exceeded the context limit, trim or preprocess inputs that were malformed, or adjust the prompt for a systematic error, then submit those as a new, smaller batch reusing the same custom_ids.

results -> partition by status
  succeeded[]  -> write to store, done
  errored[]    -> for each custom_id: fix cause (e.g. chunk oversized doc)
               -> resubmit ONLY these as a new batch

For extra savings, batch composes with prompt caching: a shared system prompt or shared context across many requests can be cached so you pay the discounted batch rate on top of cache reads. The combination is common in high-volume extraction pipelines.