Engineering
6 min
"Stream ended without finish_reason": What an OpenAI-Compatible API Should Do When the Worker Dies Mid-Stream
A customer's coding agent showed us a screenshot: Error: Stream ended without finish_reason. On our side the request row said completed. Both were wrong in their own way. Here is what happens to a streamed completion when the GPU worker's connection drops, why the common gateway behaviour turns it into a silent truncation, and the three rules we now follow: retry before the first token, an explicit error event after it, and never count a cut stream as a success.
inference APIstreamingreliability