The proxy wasn't dead, it was jammed

The client’s message said the app was stuck on “loading” and had been since about nine. The application log said something narrower, and misleading.

Redis Client Error: read ECONNRESET
  at TCP.onStreamRead (node:internal/stream_base_commons:218:20)

A connection reset reads like the far end went away. So my first theory was that Redis had died. It hadn’t — I could reach it from a shell on the same host, and redis-cli ping came back instantly.

My second theory was the firewall. That was wrong too, and I spent forty minutes proving it.

What was actually happening

Redis sat behind an nginx stream block acting as a plain TCP proxy. nginx was running. It answered on the port. It simply never completed a connection, because it had run out of file descriptors and could no longer call accept().

2026/08/29 09:14:22 [alert] 1131#1131: accept4() failed (24: Too many open files)

Errno 24. The proxy had been in that state since roughly 09:00, which matched when the client first noticed.

That distinction matters more than it sounds:

• A dead proxy refuses connections. The client fails immediately and the error points straight at the problem.
• A jammed proxy lets the kernel finish the TCP handshake and queue the connection, then never picks it up. The client believes it is connected and waits, until something upstream times out and reports a reset.

The symptom points away from the cause. That is why two reasonable theories both survived longer than they deserved.

Why it ran out

worker_connections had been set generously. worker_rlimit_nofile had never been raised to match it, so nginx was permitted to want far more connections than the kernel would let it hold open. It ran fine for months at low traffic and fell over the first time it didn’t.

The fix wasn’t the limit

Raising the descriptor limit would have worked. I didn’t do that, because the proxy shouldn’t have been in the path at all — both machines were already in the same private network and had no reason to talk through a public hop.

Since then I’ve found the same shape twice more in our own infrastructure: two services inside one private network, reaching each other by going out to the public internet and back. Each time it was invisible until something in the middle jammed.