How we batch-generate 50,000 cover images a month for Interior AI Designs
The actual pipeline behind 50k monthly room renders at Interior AI Designs: queues, FAL, idempotency keys, and the boring failure modes nobody writes about
- ↳Queue everything. Synchronous image generation at scale is a foot-gun, even at sub-second latency.
- ↳Idempotency keys per render request saved us from double-charging users when FAL retried under load.
- ↳Cloudflare R2 plus signed URLs costs us about 1/10th of what S3 + CloudFront did, and we never hit egress fees.
- ↳The expensive part of batch generation is not GPU minutes, it is moderation, retries, and the long tail of 4 percent that fail.
Interior AI Designs runs about 50,000 cover image generations a month right now. Some months it spikes to 70k when we push a new style pack. The pipeline has broken in maybe a dozen interesting ways over the last year, and I want to write down what we actually do, because every post I read on “AI image pipelines at scale” is either a vendor pitch or a toy demo with three rooms.
This is the boring infrastructure version. The version where you learn that 96% of your engineering time goes into the 4% of jobs that fail.
What a cover image actually is for us
Every user upload on Interior AI Designs gets a primary render plus 3 to 6 style variations. The primary is what we call the cover. It is the one shown in the gallery, the share card, the OG image. It has to be deterministic enough that the same room with the same prompt produces something visually consistent, but varied enough that two users redesigning a similar IKEA living room do not get the same output.
One user upload, in practice, is 4 to 7 image generations. So 50,000 covers per month is closer to 250,000 raw generations going through FAL. Plus moderation passes. Plus retries.
The stack, end to end
We run the whole thing on Cloudflare Workers, with a Postgres on Hetzner for the source of truth and R2 for storage. The queue is Cloudflare Queues. The model inference is FAL, mostly FLUX dev and a few internal LoRAs we trained on InvokeAI.
A single render request looks like this:
type RenderJob = {
id: string; // ulid, used as idempotency key
userId: string;
uploadId: string;
style: StyleId;
variant: number; // 0 = cover, 1..n = alts
inputUrl: string; // signed R2 url
promptVersion: string; // 'v7.2' so we can replay
createdAt: number;
};The id is the most important field in the whole system. We hash it into the FAL request as the idempotency key, we use it as the R2 object key, we use it for the row in renders. If anything retries, it lands on the same row. We learned this the hard way after a week in March 2024 where a Worker timeout caused us to generate (and bill ourselves for) about 8,000 duplicate images.
Why we queue everything
The naive version of this is: user uploads a photo, you call FAL inline, you return the URL. FLUX dev is fast. Sub-2-second latency. Why queue?
Because at 50k covers per month, you are at roughly 1 request every 50 seconds on average, but the distribution is awful. We get bursts of 30 to 80 concurrent renders when a TikTok hits, and we get long quiet periods at 4am ET. If you handle everything inline, your p99 latency is whatever FAL’s worst day looks like, and your error budget is whatever percentage of users are willing to refresh.
Queueing gives us four things:
- Backpressure. We can cap FAL concurrency at 40 and let the queue absorb the rest.
- Retries with backoff, without holding an HTTP connection open.
- A single chokepoint where we run moderation, log cost, and dedupe.
- A way to replay history when we ship a new prompt version.
The trade-off is the user waits. We show a progress UI that polls every 1.5 seconds. Median time from upload to cover-ready is about 11 seconds. p95 is 34 seconds. p99 is wherever FAL is having a bad day.
The moderation layer nobody talks about
Before we hit FAL, every input image goes through a moderation pass. We use a small vision model to flag obvious problems: not-a-room photos, photos of people as the primary subject, screenshots, NSFW. We reject about 2.3% of uploads at this stage.
After FAL, we run a second pass on the output. FLUX is well-behaved but it will occasionally hallucinate a person into a couch, or render text that looks like a brand logo. We reject and regenerate about 0.8% of outputs.
That 0.8% sounds small. At 250k generations a month, it is 2,000 regenerations. Each one is a real GPU cost we eat, not the user.
The retry policy that actually works
We tried exponential backoff with jitter. Standard advice. It was wrong for us, because most FAL failures cluster - when one fails, several fail, and exponential backoff just pushes the storm 30 seconds later.
What works:
const RETRY_DELAYS_MS = [800, 4000, 15000]; // three tries, then dead-letter
async function runJob(job: RenderJob, attempt = 0): Promise<Result> {
try {
return await fal.run(job);
} catch (err) {
if (attempt >= RETRY_DELAYS_MS.length) {
await dlq.send(job);
return { status: 'failed', err };
}
if (!isRetryable(err)) {
await dlq.send(job);
return { status: 'failed', err };
}
await sleep(RETRY_DELAYS_MS[attempt]);
return runJob(job, attempt + 1);
}
}The isRetryable check matters. A 429 from FAL is retryable. A 400 saying our image is too large is not. Retrying non-retryable errors is how you burn through GPU credits and end up emailing FAL support at 2am.
We dead-letter about 0.2% of jobs. A human (me, usually) looks at the DLQ once a week and decides what to do. Most of them are users who uploaded a 50MB HEIC that our resize step choked on.
R2 and the egress math
We were on S3 + CloudFront for the first six months. Egress was eating us alive. Every cover gets viewed in a gallery, shared on social, embedded in OG cards. A single popular project could rack up $30 in egress in a week.
Migrating to R2 was a weekend. R2 has zero egress fees. Storage is roughly the same price as S3. We saved about $1,400 a month on bandwidth alone, which more than pays for the engineering time it took to move.
The gotcha: R2’s API is S3-compatible but not perfectly. Multipart uploads behave slightly differently. Signed URL expirations cap at 7 days. We had to rewrite our share-card generator to use 7-day URLs and refresh them rather than 30-day ones.
What I would do differently
If I were starting today, I would not build my own queue layer. I would use Inngest or Trigger.dev from day one. We built ours on Cloudflare Queues and it works, but every observability feature I now want (per-step retry counts, fan-out graphs, replay-from-step) we have had to build ourselves.
I would also start with idempotency keys before I needed them. We retrofitted them after the duplicate-billing incident. Retrofitting idempotency into a live system is genuinely awful.
The interesting failure modes are never the model. The model is fine. The interesting failures are the boring distributed-systems stuff: a webhook fires twice, a queue ack gets lost, a Worker hits its CPU budget mid-upload. The same problems Stripe was writing about in 2014. Image AI did not change that.
If you are building something similar and want a second pair of eyes on your pipeline, come say hi.
Common questions
▸Why FAL over Replicate or running your own GPUs?
We benchmarked all three in early 2024. FAL was consistently 2 to 4x faster for FLUX dev at the time and had better autoscaling under bursts. Replicate works fine but cold starts hurt when traffic is bursty. Running our own H100s on Lambda Labs or Hetzner would be cheaper per generation at steady state, but the ops overhead for our team size was not worth it until we are at 200k+ covers a month.
▸How do you keep prompt costs predictable?
Two things. We cap per-user generations per day (10 free, 100 on paid) so a runaway script cannot empty our balance. And we log FAL request cost per render to Postgres so we can see margin per user in real time. If margin drops below a threshold, alerts fire. Most cost surprises come from regenerations after moderation failures, not from base traffic.
▸What does the human moderation queue look like?
There is no human moderation queue for normal flow. The vision model handles pre and post filtering. Roughly 30 to 50 outputs per week get appealed by users, and we review those manually. We deliberately bias the post-generation moderator toward false positives because regenerating is cheaper than shipping a bad cover to a paying customer.
▸Why Cloudflare Workers instead of a Node server?
Workers, Queues, and R2 sit in the same network so the latency between queue consumer, storage, and HTTP egress is negligible. We do not pay for idle. Free tier covers our staging environment entirely. The constraint is the 30 second CPU limit per request and lack of persistent connections, which is why heavy work goes through the queue rather than inline.
▸How do you handle prompt versioning when you change the cover style?
Every render row stores a promptVersion string. When we ship a new version, old covers keep their old promptVersion. If a user wants to regenerate with the new style, they explicitly opt in and we create a new render row, never overwriting. This means we can roll back a bad prompt change by flipping which version we serve, without losing any history.
- [1]Cloudflare R2 pricingdevelopers.cloudflare.com
- [2]FAL documentationfal.ai
- [3]Cloudflare Queues documentationdevelopers.cloudflare.com
- [4]Stripe API idempotencydocs.stripe.com
Related posts
How we batch-generate 50,000 cover images per month for Interior AI Designs
The actual queue, the actual costs, the actual failure modes behind running 50k FLUX renders a month for Interior AI Designs without lighting our bank account on fire.
Five product decisions that doubled Photo AI Studio retention
What actually moved the needle on Photo AI Studio's D7 retention. Pricing, onboarding, model choice, refund flow, and a push notification we almost shipped wrong.
Cloudflare Turnstile vs reCAPTCHA: why we migrated and what it cost us
We pulled reCAPTCHA out of three production apps and replaced it with Cloudflare Turnstile. Here's what broke, what it saved, and where Turnstile still falls short.