A customer running a coding agent against our API sent us a screenshot. The agent had just found a precedence bug in a shell script, printed the diff, and then:
Error: Stream ended without finish_reason
On our side, the request row for that call said completed, 0 tokens,
13.7 seconds. Two systems looking at the same event and both reporting it
wrong: the client saw a failure it could not explain, our books saw a
success that never happened.
This post is what actually happened between the two, and the three rules we changed in our gateway because of it. They apply to anyone proxying streamed completions from a pool of workers: an OpenAI-compatible gateway, a router in front of vLLM or llama.cpp, an internal load balancer.
The event
A streamed chat completion is a Server-Sent Events response. The worker
sends data: lines, one JSON chunk each; the last chunk carries
finish_reason and, with stream_options.include_usage, the token counts;
then data: [DONE]. The client knows the answer is complete when it sees
those two things.
Here is the timeline of the request in the screenshot, from our logs:
| time | event |
|---|---|
| 0.0 s | request accepted, routed to a two-GPU worker running a 27B model |
| 0.0 to 13.7 s | prefill of a long prompt; no token had been produced yet |
| 13.7 s | the worker's WebSocket tunnel to our hub drops: close 1006, unexpected EOF |
| 13.7 s | the hub ends the proxied response cleanly |
| 13.7 s | the gateway sees the upstream stream end without [DONE]; it logs that and closes the client's stream |
| 18.7 s | the worker reconnects; its model never stopped running |
Nothing was wrong with the GPU or the model. A five-second network blip on the provider's side, during the one window where no byte had reached the client yet.
Why the client saw what it saw
The gateway had already committed to the stream: it had sent the client
200, Content-Type: text/event-stream, and then nothing. When the
upstream closed, the gateway closed too. From the client's side, the stream
ended: no chunk with finish_reason, no [DONE]. The OpenAI client
libraries treat that exactly as they should, as an error. Hence the
message.
Why our books said "completed"
The gateway's accounting was written for the common case. A stream that
ends on a transport error is recorded as an error; a stream that ends
cleanly is recorded as completed, with whatever usage was received,
which here was none. A clean end without [DONE] was logged as a warning
and otherwise counted as success. Over a day that is one line in a log
nobody reads, and a success rate that is a little too good.
Rule 1: before the first token, retry
At 13.7 seconds, zero bytes of answer had been sent to the client. Nothing that happens to the upstream at that point needs to be visible to the caller. A non-streaming request in our gateway already retried worker faults on another worker; the streaming path did not, because it "committed" to streaming as soon as the worker returned 200.
Now it commits later. The gateway opens the upstream stream and reads until
the first data: line before sending the client anything. If the
upstream ends or fails before that line, the request is retried on another
worker like any other worker fault, and the client never learns. The
first-byte watchdog still bounds the wait, so a worker that produces
nothing for minutes does not hold the client forever.
For the request in the screenshot, this alone would have turned a failure into a 20-second answer from a different node.
Rule 2: after the first token, say why
Once tokens have been sent, retrying is not an option: the client has half an answer, and a second worker would produce a different one. What the gateway can do is tell the truth. Two changes:
-
The hub speaks. When a worker's tunnel drops mid-request, our hub used to close the proxied stream silently. It now writes one last event before closing:
data: {"error":{"message":"agent disconnected mid-request","type":"server_error","code":"worker_disconnected"}} -
The gateway passes it on, or writes its own. An error event from upstream is forwarded as-is and the stream ends there. A stream that ends without
[DONE]and without an error event gets a synthetic one,stream_interrupted, before the close. Either way the client's SDK raises a real error with a real reason, instead of "ended without finish_reason".
We deliberately do not send [DONE] after an error event. [DONE]
means the answer completed; it did not.
Rule 3: a cut stream is an error in the books
The request row is now marked error in all three cases: transport error,
upstream error event, clean end without [DONE]. The message says which,
with the timing. Anything else inflates the success rate with responses
that no client accepted.
This is the rule that matters most over time. The other two fix the customer's experience; this one fixes ours. A dashboard that shows 99.9% success while callers see cut streams sends the engineer to look at the wrong thing.
The detail that made retry safe
Reading the first data: line before committing has one subtlety: the
line has to be handled exactly as it would have been inside the stream,
including time-to-first-token, usage parsing and the model-name rewrite,
and the lines that preceded it (SSE comments, blank keep-alives) have to be
written to the client in order. Our implementation buffers those, hands the
first data line to the same per-line handler the loop uses, and continues.
It is thirty lines, and the unit test that covers the error-event detection
also checks that a chunk whose content contains the word "error" is
not mistaken for one.
What to check in your own gateway
- Send a streaming request with a prompt long enough that prefill takes several seconds, and kill the worker's connection during that window. Did the client get an answer, or an error?
- Kill it after a few hundred tokens. Did the client get an explicit error event, or a stream that just stopped?
- Look at the request in your metrics afterwards. Success, or error?
If any of the three answers is the wrong one, the fix is above. Ours was found by a screenshot; a test would have been cheaper.
