One connection in five hung for five minutes instead of loading

About one connection in five to any Think site was accepted and then left hanging for five minutes before giving up, instead of loading or failing quickly. It came and went, and got steadily worse over a few hours until nineteen connections in twenty were affected.

What was happening

The platform's web entry point keeps one listening slot per worker process. A host limit on how many tasks a single service may run was set far too low, so workers were killed off as soon as the service got busy — and once killed they were never replaced. Their listening slots stayed behind, still being handed new connections that nothing was left to answer. The connection opened, the browser said hello, and then nothing ever came back.

The share of hung connections matched the share of abandoned slots exactly, which is how the cause was confirmed.

Why nobody noticed

Nothing detected it. The automated health check runs on a separate port and never touches the affected path, so every check reported the service healthy the entire time. A hung connection also never reaches the destination, so it leaves no trace in any log — the usual dashboards stayed clean while a fifth of traffic was stalling.

Retries hid the rest. A browser or a build job usually succeeded on a second attempt, so the visible symptom was only ever intermittent slowness. Automated jobs that do not retry failed outright, which is what finally exposed it.

The fix

The host limit was wrong because of how the machines are built: they start with a small amount of memory and receive the rest moments later, and the limit is calculated once, early, from what was present at the time. Fifteen of eighteen machines were affected.

The limit is now set explicitly on all of them, folded into the configuration baseline so it survives rebuilds, and every affected component was restarted onto the corrected value.