Sooner or later every Shopify project hits the same wall: you need to touch thousands of products, images, or variants at once, and the obvious approach — loop and call the API for each one — is far too slow and burns through the rate limit in seconds. Shopify’s answer is the Bulk Operations API, and using it well is the difference between a job that finishes in minutes and one that never finishes at all.
Why the naive loop fails
The Admin API is rate-limited by design. Iterating over 50,000 products with one mutation each means 50,000 throttled requests, constant backoff, and an operation that runs for hours while you babysit it. It is the wrong tool for bulk work.
Bulk queries: let Shopify do the export
For reading large datasets, a bulk query runs asynchronously on Shopify’s side and hands you a single downloadable file when it is done. You submit the query, poll for completion, then stream the results — no pagination loop, no rate-limit dance:
mutation {
bulkOperationRunQuery(
query: """
{ products { edges { node { id title
images { edges { node { id altText } } } } } } }
"""
) {
bulkOperation { id status }
userErrors { field message }
}
}
The result arrives as JSONL — one JSON object per line — which is ideal for streaming through without loading gigabytes into memory:
await foreach (var line in ReadJsonlAsync(resultUrl))
{
var node = JsonSerializer.Deserialize<ProductNode>(line);
// process one record at a time, constant memory
}
Bulk mutations: write at scale
Writing works the same way in reverse. You upload a JSONL file describing every change, then run a single bulkOperationRunMutation that applies them asynchronously. Thousands of updates become one submitted operation instead of thousands of live calls, and Shopify processes them at its own sustainable pace.
Idempotency and resumability
A job over tens of thousands of records will occasionally be interrupted. We record progress as we go so a restart resumes from where it stopped rather than redoing everything, and we design each update to be safe to reapply. Re-running the job never double-applies a change — it simply finishes what was left.
Keeping the merchant informed
Bulk work is asynchronous, so the interface has to reflect that. Rather than a frozen spinner, we poll the operation status and surface real progress — records processed, estimated completion — so a large job feels controlled instead of broken.
What this buys you
- A job that would have meant tens of thousands of throttled API calls collapses into a single asynchronous operation Shopify runs on its own pipeline.
- Rate-limit throttling stops being the bottleneck for bulk work.
- Interrupted runs resume cleanly, with no duplicated work.
Takeaway
When a Shopify task involves thousands of records, stop thinking in individual API calls and start thinking in bulk operations. Offloading the heavy lifting to Shopify’s asynchronous pipeline, streaming JSONL results, and building in resumability turns an impossible loop into a routine background job. This is exactly the engineering behind processing catalogs at scale — something we do regularly, including in our own published apps. Get in touch if you have a bulk Shopify job that will not finish.