How we batch-generate 50,000 cover images per month for Interior AI Designs
The actual queue, the actual costs, the actual failure modes behind running 50k FLUX renders a month for Interior AI Designs without lighting our bank account on fire.
- ↳Cheap models for drafts, expensive models for finals. Mixing FLUX schnell and FLUX dev cut our per-image cost roughly in half.
- ↳A queue is not optional at 50k/month. We run everything through Cloudflare Queues with idempotency keys so retries do not double-bill us.
- ↳Most failures are not the model. They are prompt edge cases, NSFW false positives, and webhook drops.
- ↳Storing the prompt, seed, and model version with every render is the difference between debugging in 5 minutes and 5 hours.
Interior AI Designs renders somewhere between 45,000 and 55,000 cover images a month. Listing hero shots, style variations, before/after pairs, social previews. The pipeline runs on FAL, Cloudflare Queues, and R2, glued together with a TypeScript worker that I have rewritten three times since January.
This is the post I wish I had when we started. Not the architecture diagram version. The version where I tell you what broke.
The shape of the workload
A single user action (“restyle this room as Japandi”) fans out into 4 to 12 images. Different angles, different palettes, a couple of safety regens. So 50k cover images is really closer to 8k user jobs, each producing a batch.
The traffic is bursty. Mondays and Thursdays do roughly 2.3x the volume of Saturdays. Realtors batch-upload listings before their week starts and before their weekend showings. Knowing that shape changed how we provision: we do not. We let the queue absorb it.
Here is the contract every job satisfies:
type RenderJob = {
jobId: string; // ULID, also the idempotency key
userId: string;
sourceImageR2Key: string;
prompt: string;
negativePrompt?: string;
model: "flux-schnell" | "flux-dev" | "flux-pro";
seed: number;
variant: number; // 0..n within a batch
webhookUrl: string;
createdAt: number;
};The jobId is the only thing that matters. Everything downstream is keyed on it. If a worker crashes mid-flight, the retry produces the same output path, the same R2 key, the same row. No duplicates billed.
Why FAL, and what it actually costs
We tried Replicate, Together, and self-hosting FLUX on a pair of A100s through Hetzner for six weeks. FAL won on three things: cold-start latency (consistently under 2s for schnell), webhook reliability, and the fact that their billing dashboard does not lie to me.
Self-hosting on Hetzner was cheaper per image on paper, around 40% cheaper at full utilization. The problem is we are not at full utilization. We are at 18% average GPU utilization with spikes to 95%. Paying for idle A100s at $1.80/hr each made the math worse than FAL by the time I added on-call rotation for the GPU box falling over.
Current blended cost per final image, including drafts we throw away: about $0.011. That is across all three FLUX variants. Schnell does the cheap drafts at roughly $0.003 each, dev does the finals at roughly $0.025, pro only shows up for the marketing landing page hero generations.
The queue is the product
Everything goes through Cloudflare Queues. One producer, three consumer workers, separated by priority:
render-interactive: user is staring at a loading spinner. Target p95 under 8 seconds.render-batch: bulk listing imports. Target p95 under 4 minutes.render-retry: failed jobs, exponential backoff, max 4 attempts.
The consumer is dumb on purpose:
export default {
async queue(batch: MessageBatch<RenderJob>, env: Env) {
for (const msg of batch.messages) {
const job = msg.body;
const existing = await env.DB.prepare(
"SELECT output_key FROM renders WHERE job_id = ?"
).bind(job.jobId).first();
if (existing?.output_key) {
msg.ack();
continue;
}
try {
const result = await fal.run(job.model, {
input: buildInput(job),
webhookUrl: `${env.WEBHOOK_BASE}/fal/${job.jobId}`,
});
await recordPending(env, job, result.request_id);
msg.ack();
} catch (err) {
if (isRetryable(err)) msg.retry({ delaySeconds: backoff(msg.attempts) });
else { await markFailed(env, job, err); msg.ack(); }
}
}
}
}Idempotency check first, always. FAL request goes out with a webhook pointing back at a Worker route that writes the final R2 key. The user-facing API polls our D1 table, not FAL.
Things that broke that I did not expect
Webhook drops. FAL is reliable but not perfect. We were losing somewhere around 0.4% of webhooks in March. The fix was a sweeper job that runs every 90 seconds, picks up any pending rows older than 2 minutes, and polls FAL directly for status. That number is now under 0.01%.
NSFW false positives on bedrooms. FLUX dev’s safety classifier flagged maybe 3% of bedroom renders. A bed, a pillow, no people. The model still said no. We route bedroom prompts through a slightly different model config and append explicit “unfurnished mannequin-free interior photography, no people” to the negative prompt. False positive rate dropped to under 0.4%.
Prompt drift across model versions. When FAL updated their FLUX dev endpoint in April, the same seed produced visibly different output. Saturation shifted warmer. We now pin model versions explicitly and store the version string with every render. If we want to regenerate a user’s gallery for a re-export, we can. Without that, we could not.
R2 list operations are not free. I built the first version of the gallery by listing the user’s R2 prefix. At a few hundred users this was instant. At 12,000 users it was costing us more in Class A operations than the actual rendering. Everything is indexed in D1 now, R2 is treated as dumb storage.
How we keep prompts sane
Every prompt is composed from three layers: a style template (Japandi, Mid-Century, Scandi, etc), a room-type modifier (living room, bedroom, kitchen), and a user-tunable slider for “how much to change.” The style templates are versioned in Git and have unit tests that render against a fixed seed and compare to a reference image with a perceptual hash.
When a designer on the team wants to add a new style, they open a PR. CI renders 12 test images. If the hash drift on existing styles is above a threshold, the PR fails. This has caught two regressions where a small wording change wrecked unrelated styles.
What I would do differently
I would have written the sweeper job on day one instead of day 60. I would have stored the full prompt, seed, model, and version from the very first render instead of bolting it on at 8,000 users. And I would have skipped the Hetzner detour entirely. Six weeks I am not getting back.
If you are running an AI product that fans out into a lot of generations and the bill is getting weird, or your queue is held together with hope, we do this for a living. Tell us what you are shipping.
Common questions
▸Why not self-host FLUX if you are doing 50k images a month?
We tried for six weeks on two A100s via Hetzner. Per-image cost was about 40% lower at full utilization, but our actual utilization averaged 18% with spikes to 95%. Once you add the on-call burden of a GPU box that will eventually fall over at 3am, FAL came out cheaper and saner. The break-even point for us would be closer to 200k images a month with steadier load.
▸How do you avoid double-billing on retries?
Every job carries a ULID we generate before it hits the queue. The consumer checks D1 for an existing output key under that job ID before calling FAL. If it exists, we ack and move on. The same ID is used as the R2 output path. A retry that races with a successful first attempt simply overwrites identical bytes, and FAL never gets called twice.
▸What does the blended cost per image actually include?
Roughly $0.011 per final delivered image. That includes draft generations on FLUX schnell that we discard, the final FLUX dev render, R2 storage and egress, Cloudflare Workers requests, and D1 writes. It does not include the fixed cost of the team or the safety regens we eat when the NSFW classifier misfires, which add about 2% overhead.
▸Why Cloudflare Queues instead of SQS or BullMQ?
The whole stack runs on Workers and R2 already. Adding SQS meant another IAM surface, another billing account, and a network hop. Cloudflare Queues has lower throughput ceilings than SQS, but at 8k batches a day we are nowhere near them. BullMQ would have meant running Redis, which would have meant running a server, which defeats the point.
▸How do you debug when an image looks wrong?
Every render row stores the full prompt, negative prompt, seed, model name, model version string, and source image hash. Given a job ID I can reproduce the exact render in about 30 seconds by replaying it through a CLI script that hits FAL with the same input. Without that history, regressions from upstream model updates would be impossible to diagnose.
- [1]Cloudflare Queues documentationdevelopers.cloudflare.com
- [2]FAL FLUX modelsfal.ai
- [3]Cloudflare R2 pricingdevelopers.cloudflare.com
Related posts
How we batch-generate 50,000 cover images a month for Interior AI Designs
The actual pipeline behind 50k monthly room renders at Interior AI Designs: queues, FAL, idempotency keys, and the boring failure modes nobody writes about
Five product decisions that doubled Photo AI Studio retention
What actually moved the needle on Photo AI Studio's D7 retention. Pricing, onboarding, model choice, refund flow, and a push notification we almost shipped wrong.
Cloudflare Turnstile vs reCAPTCHA: why we migrated and what it cost us
We pulled reCAPTCHA out of three production apps and replaced it with Cloudflare Turnstile. Here's what broke, what it saved, and where Turnstile still falls short.