four endpoints, eight megabytes each, and a stranger with a for loop

the public stats page was the slow one. 480 to 600 milliseconds solo, 13.8 seconds at the 95th percentile under fifty concurrent users, three sequential scans of a 200,000-row table per request. it was the headline finding of the torro.cat ↗ pre-launch audit and it deserved to be.

the share-card endpoints were fast. they render a PNG and hand it back, no database scan, nothing clever. i nearly skipped them.

then i multiplied two numbers together.

a rate limiter counts requests. nobody was counting work.

torro.cat has four public routes that return a generated image — a personal result card, a “wrapped” recap, a reveal card and a press-kit one-pager. all four are 1080×1920, the instagram stories aspect ratio, because the entire point of the site is that you screenshot your nougat ranking and post it.

here is the arithmetic i should have done on day one. an RGBA image is four bytes a pixel:

1080 × 1920 × 4 = 8,294,400 bytes ≈ 8.3 MB

that’s one buffer, allocated for the duration of one render, plus a CPU core pegged while the thing is painted and PNG-encoded. and there were four unauthenticated routes that would each do it on request, with no bound of any kind between them.

so: ten concurrent requests is 83 MB and ten saturated cores. fifty is 415 MB. this runs on one small VPS. a for loop and a public URL is the entire attack, and it isn’t really an attack — it’s what happens if one of these cards actually does what it was built to do and gets shared somewhere busy.

i had spent the previous week fixing a rate limiter , and i want to be precise about why that didn’t help here, because i initially assumed it would. the per-ip limiter caps how many requests an address makes per minute. it says nothing about how much work a request causes. a hundred requests a minute is a perfectly reasonable limit for an endpoint that costs a database lookup, and a catastrophic one for an endpoint that costs eight megabytes and a core. the limiter was correctly configured and completely irrelevant.

the resource you have to bound is the scarce one. on this box that’s cores and RAM, not requests.

a counting semaphore, and the decision to shed rather than queue

the fix is a buffered channel used as a counting semaphore, at package level, shared by all four routes:

internal/sharecard/canvas.go

// renderSlots is the counting semaphore: a buffered channel with one slot
// per allowed concurrent render. A send acquires a slot, a receive releases
// it. Sized to GOMAXPROCS (the renders are CPU-bound) with a floor of 2 so
// even a single-core box still serves two callers rather than fully
// serializing.
var renderSlots = make(chan struct{}, maxRenderConcurrency())

func maxRenderConcurrency() int {
	if n := runtime.GOMAXPROCS(0); n > 2 {
		return n
	}
	return 2
}

sizing it to GOMAXPROCS is the whole idea: the work is CPU-bound, so the useful number of simultaneous renders is the number of cores. more than that doesn’t render anything faster, it just holds more 8.3 MB buffers alive at once. the floor of 2 exists because fully serializing on a single-core box makes one slow client block another for no benefit.

one semaphore for all four routes, not one each. four separate caps of N is a cap of 4N, which is not a cap — it’s the same bug with extra steps. the constrained resource is the machine, so the bound belongs to the machine.

the acquire is where the actual design decision lives:

// RenderSlotWait bounds how long a caller waits for a render slot before
// giving up and shedding load. Kept short so a burst sheds fast instead of
// queueing requests that would themselves time out.
const RenderSlotWait = 250 * time.Millisecond

func TryAcquireRenderSlot(parent context.Context) bool {
	ctx, cancel := context.WithTimeout(parent, RenderSlotWait)
	defer cancel()
	select {
	case renderSlots <- struct{}{}:
		return true
	case <-ctx.Done():
		return false
	}
}

three things in nine lines, and each was a choice:

it derives from the request context, so a client that has already hung up stops waiting for a slot immediately instead of holding a place in a queue for a response nobody will read.

it waits 250 milliseconds and then gives up. an unbounded wait turns a memory problem into a latency problem and calls it fixed: requests pile up, each one holding a connection, and they all eventually blow through the server’s write timeout and return truncated errors anyway. you have not shed the load, you’ve deferred it and made it worse. a short wait absorbs a genuine micro-burst — which is most of them — and refuses everything past that.

it returns a bool rather than an error, which forces the caller to branch. and the caller sheds properly:

internal/http/sharecard_handler.go

// Cap concurrent renders (each allocates a large RGBA and pegs a core):
// shed load with 503 rather than piling up when the cap is saturated.
if !sharecard.TryAcquireRenderSlot(r.Context()) {
	logger.Warn("[Handler - ShareCard] Render slots saturated; shedding request.")
	w.Header().Set("Retry-After", "1")
	http.Error(w, "Service Unavailable", http.StatusServiceUnavailable)
	return
}
defer sharecard.ReleaseRenderSlot()

503 with Retry-After, not 429. the distinction matters and i had to look it up: 429 Too Many Requests says you have sent too many, which is a statement about the client. 503 Service Unavailable says i am temporarily out of capacity, which is a statement about the server. a first-time visitor who arrives during a burst has done nothing wrong, and telling them they’re rate-limited is both rude and false.

note the asymmetry that the API is shaped to enforce: acquire returns a bool, and only a true may be paired with a release. a ReleaseRenderSlot() called after a failed acquire would receive from the channel and hand out a slot nobody acquired — permanently widening the cap by one, every time it happened, until the bound is meaningless. that’s why the doc comment on the function says it in capitals and why the defer sits after the branch rather than before it.

the part i didn’t expect: bounding concurrency is admitting you have it

here’s the thing that made this more interesting than “add a semaphore.” for the entire life of this code, renders had been effectively serial in my head — one card at a time, one request at a time, nothing to think about. deliberately allowing GOMAXPROCS concurrent renders is a decision to have real, simultaneous, multi-goroutine execution through a package that had never been audited for it.

so everything the render path touches has to be sorted into “safe to share” and “not,” and the answer is different for the two things it shares.

the PNG encoder is fine:

// png.Encoder is safe for concurrent use when only CompressionLevel is
// set (Encode never mutates it), so one package-level value serves all
// renders.
var pngEncoder = png.Encoder{CompressionLevel: png.BestSpeed}

the font faces are emphatically not. golang.org/x/image/font ↗ says so in the interface’s own documentation, and the reason is the useful part:

a Face is not safe for concurrent use by multiple goroutines, as its methods may re-use implementation-specific caches and mask image buffers.

a Face looks immutable. it’s a font at a size; conceptually it’s a value. but rasterizing a glyph is expensive, so implementations keep a scratch buffer and a glyph cache inside, and two goroutines drawing text through one Face will interleave writes into the same mask buffer. the failure mode is not a crash — it’s a card with a corrupted glyph on it, occasionally, under load, which is about the worst way for a bug to present itself.

so the face cache is per-canvas and a canvas is per-request:

// So: a canvas, and its face cache, must never be shared across
// goroutines. newCanvas is called once per Render call (i.e. once per HTTP
// request), and canvases are never stored anywhere longer-lived than that
// - the underlying *sfnt.Font values are parsed once at package init and
// ARE safe to share, since building a Face from them is cheap and every
// canvas builds its own.
type canvas struct {
	img   *image.RGBA
	faces map[faceKey]font.Face
}

three tiers, and you need all three: the parsed *sfnt.Font is immutable and shared once at init; the Face built from it is mutable and lives exactly one request; the cache holding faces belongs to the canvas that owns them. i’d have got this wrong if the semaphore hadn’t made me ask.

a concurrency limit above 1 is a promise that the code underneath is concurrency-safe. the semaphore doesn’t make anything safe. it’s the thing that makes the unsafety reachable.

while i was in there: compression was two thirds of the cost

the audit measured where render CPU actually went, and png.DefaultCompression was about 68% of it. that’s the majority of the work, spent shrinking an image that exists to be posted to instagram and forgotten:

// It uses BestSpeed compression on purpose: png.DefaultCompression
// accounted for ~68% of a render's CPU, and share cards are ephemeral
// social images regenerated on demand, so trading a modestly larger byte
// size for markedly less CPU is the right call.

this is the cheapest fix in the whole audit and it interacts directly with the semaphore: cutting per-render CPU means each slot frees sooner, which means the same cap serves more requests per second. bounding a resource and reducing what you spend inside the bound are the same project.

it’s also a fair trade only because of what these files are. a bigger PNG costs bytes on a CDN-less box, and if these were assets served a million times from cache i’d want them small. they’re generated per request, viewed once, and thrown away.

what i rejected

• a bigger box. it’s four unauthenticated endpoints with no upper bound; the machine size just determines which number of concurrent requests kills it. an unbounded resource is unbounded on any hardware.
• an unbounded queue. already covered, but it’s the most tempting wrong answer, because it makes the error rate go to zero in testing and turns the failure into timeouts in production.
• caching the rendered cards. genuinely good and genuinely orthogonal — every card is personalised to a cookie-identified user, so the hit rate would be near zero on exactly the traffic pattern i’m defending against, and a cache is not a bound. cache to be fast; cap to survive.
• making the renders asynchronous — enqueue, return a job id, poll. correct for a heavier pipeline and absurd here: the client is an <img> tag in a share sheet.

lessons learned

• rate limits bound requests; they do not bound work. if one request can cost eight megabytes and a core, “100 requests/minute” is a limit on the wrong axis entirely. ask what the scarce resource is and put the bound on that.
• one cap for the whole machine, not one per route. four routes with a cap of N each is a cap of 4N. the resource is shared, so the semaphore has to be.
• shed, don’t queue. a short bounded wait absorbs a real burst; an unbounded one converts an OOM into a pile of timeouts and hides the problem until it’s a worse one. and shed with the honest status code — 503 blames the server, 429 blames a visitor who did nothing wrong.
• allowing N > 1 is a claim about every line underneath. the semaphore didn’t introduce a data race; it made an existing latent one reachable, and forced me to actually sort shared state into immutable-and-shared, mutable-and-per-request, and the cache that owns the mutable thing. font.Face looked like a value and wasn’t.
• measure where the CPU goes before optimising the part you assume is slow. i’d have guessed the drawing. it was the compression, by two to one.

that closes the pre-launch audit: an elo race that ate 79% of every vote , a rate limiter a rotated header walked through , a stats page that was a denial of service i hosted myself , and four endpoints that would hand a stranger the whole box. none of them were visible from the code. all of them were visible the moment i pointed load at it and watched.

next: something much bigger and much older. modelling eighty years of race results, where the points system changed, the classes were renamed and merged, and “a win” in 1949 and “a win” in 2026 are not the same row — and every query that pretends otherwise returns a confidently wrong answer.