Chasing an Elusive Rails SSE Hang

Just after midnight on a Saturday in June, the AI registry consultant in our iOS app stopped serving traffic. It happened twice more over the next eleven days. Each time the servers behind it were up and listening, but unable to process requests. We spent two weeks chasing it as a resource leak without traction, and only made real progress once we could reproduce it on demand. A thread dump off a hung production worker finally showed the cause: a race condititon in Rails' own ActionController::Live. That fix is now in Rails.

What we saw

The assistant streams its responses to the app over Server-Sent Events, a protocol that holds a single HTTP response open so the server can push text to the client as it is produced instead of returning once and closing. A request comes into Rails, Rails opens a long-lived SSE connection back to the phone, and a second thread pumps tokens from our agent backend into that connection as they arrive. Those endpoints run on their own Puma fleet: multiple processes with multiple threads. ActionController::Live pins one worker for the entire life of a stream, which caps us at a fixed number of concurrent conversations fleet-wide without autoscaling.

Shortly after midnight Eastern on a Saturday, every one of those servers stopped serving traffic over about 35 minutes. The load balancer had nothing healthy to route to and returned 503s. Nothing crashed. Nothing restarted. Nothing logged anything unusual. Recovery required a human restarting the servers.

It happened again the next night at nearly the same wall-clock time, and again a week later - always 17 to 20 hours after the most recent deploy.

Two weeks of chasing a leak

A failure that arrives a fixed number of hours after every restart is what a slow resource leak looks like, so that is what we went after first. We started with puma_worker_killer, which recycles worker processes on a timer. We ran it in staging for several hours configured to fire every 10 minutes so we could watch many cycles rather than a handful, confirmed it was safe, then rolled 6-hour rolling restarts out to production and babysat the first cycle that evening. We confirmed the server restarts were working as designed. The outage recurred anyway, well inside the six-hour window. Adding capacity bought time, but not a fix.

The result was negative, but it helped us triangulate the cause. Every guardrail we shipped worked precisely as designed, and the failure kept arriving on schedule regardless. Recycling workers every six hours does not help if they wedge faster than that under real traffic.

Getting it to happen on demand

So we stopped waiting on production. Staging runs a single SSE server, which meant we only had to saturate one. We opened ten concurrent conversations through the admin UI and the server went down completely: health checks failed, it never came back, and an eleventh conversation had nothing left to connect to. That was the exact production failure, on demand, in an environment where we could take our time.

The thread dump

With a reproduction in hand we could stop reasoning about the system from the outside. We attached to a hung server in production and captured a full thread dump from the wedged Puma workers.

Every one of them was blocked in the same place: SizedQueue#pop, inside ActionController::Live::Buffer#each_chunk. One stack turned two weeks of infrastructure investigation into a specific, readable bug in the framework.

Every abandoned conversation was permanently consuming one of a fixed pool of workers. They accumulated fastest overnight and over long weekends, when no deploys were incidentally recycling processes.

The race condition, and the fix

Rails hands a streaming response between two threads through a small blocking queue. A writer thread puts chunks in as they are produced; a reader thread pulls them out and sends them down the wire, and when the queue is empty it parks rather than polling, staying parked until something is pushed. To end a stream cleanly the writer pushes one final sentinel value, which the reader takes as its cue to unwind and hand the worker back. That sentinel is the only thing that ever wakes the reader. When a client disconnects mid-stream, though, Rails takes a different exit path: it marks the stream as aborted and empties the queue of anything still waiting to be sent, on the reasonable theory that there is no longer anyone to send it to. What it does not do is push the sentinel, and the reader never checks the aborted flag because it is not awake to check anything.

There are two ways this fails. In the first, the reader is already parked on an empty queue when the disconnect arrives. The abort clears nothing and pushes nothing, so the reader never wakes. In the second, the two exit paths race: the clean shutdown pushes its sentinel, and microseconds later the abort's clear sweeps it back out - before the reader has been scheduled to see it. This is what a fast client disconnect produces, and it's the more common case in practice.

Either way the writer believes the stream is torn down while the reader stays blocked forever on a message that will never arrive, and because that reader is the Puma worker, the worker is gone for the life of the process.

The fix is to make the abort path finish the job the clean path already does: empty the queue, then push the sentinel and wake anyone waiting on it. The reader unblocks, sees the terminator, and unwinds normally. It cannot stall on a full queue, because the clear immediately before it guarantees room, and if the clean path also pushed a sentinel the extra one is harmless since the reader stops at the first and never looks again. We shipped it as a local patch that afternoon, confirmed it in staging, and the outages stopped.

Getting it into Rails

None of this was specific to our application. Any Rails app streaming over ActionController::Live with a low thread count is exposed, and it fails silently - no exception, no log line, just a worker that never comes back. So we wrote it up properly: rails/rails#58047 documents both race condititon scenarios, and rails/rails#58048 carries the fix plus two regression tests, one for each interleaving. Both tests hang on main and pass with the change. Aaron Patterson merged it a few hours after it was filed. It is targeted for Rails 8.2, so we get to delete our patch at the next bump.

What's next

We added Puma worker metrics - busy versus idle workers, backlog, request queue time - so a wedged worker shows up on a dashboard long before it takes anything down. Longer term we are moving the streaming path off Rails entirely, with the app connecting directly to our agent backend using a conversation-scoped JWT.