A truncated stream is not a finished answer
BuilderHelp and RuleCaddie stream model output with the Vercel AI SDK. Streaming makes latency feel workable, but it also invents a new failure mode: the connection closes with text already on screen and no signal whether that text is complete. A spinner that disappears is not a completion marker. If you do not label truncated output as truncated, users will treat a half-written citation or owner update as finished work.
HTTP 200 is not done
Once the server returns 200 and starts a token stream, status codes cannot tell the client that something went wrong later. The model can stall on a tool call, the proxy can drop the socket, or the user can hit Stop. In every case the response that already rendered looks like ordinary assistant text unless you give it a different state.
We treat streaming as a small state machine: waiting for first token, streaming, tool work in progress, completed, incomplete, errored. Completed is only set when the stream ends with an explicit finish event from the SDK — not when the ReadableStream closes for any reason. Incomplete and errored keep whatever tokens arrived and attach a reason the UI can show.
That distinction matters more for structured product surfaces than for chat toys. An owner update that cuts off before the schedule section, or a rules answer that stops before the citation, is worse if it looks polished and final.
Keep the partial on purpose
When someone stops a generation, they often already saw the part they needed. Clearing the bubble forces a full regenerate and throws away useful draft text. The same is true for a network drop: deleting half an answer teaches people not to trust the product when the network flaps.
The rule we ship with is simple. On cancel or disconnect, leave the partial visible, mark the message incomplete, re-enable the input, and offer retry or continue from that point when the API supports it. Stop must call AbortController and also tell the server to halt generation — a button that only hides the spinner still burns tokens and can finish writing into a message the user thinks they cancelled.
For screen readers, a polite live region can announce that generation stopped or failed. Do not dump every token into a live region; announce phase changes and completion, not the character stream.
Partial JSON is not a preview you can trust
Several BuilderHelp features do not stream free prose into a chat bubble. They stream toward a structured object — matched invoice lines, a schedule tweak, a drafted owner update with sections. Mid-stream JSON is incomplete by definition. Calling JSON.parse on each chunk will throw; forcing a UI render from a half-closed object will flash wrong fields.
Accumulate the buffer. If you show a live preview, use a tolerant parser that only promotes fields that have fully closed, or render a prose draft until the stream finishes and the schema validates. Commit to application state only after validation succeeds. The incomplete label applies here too: a form half-filled from a truncated stream should not look like a saved draft.
Tool calls need the same honesty. While retrieval or a function is running, show that phase as status — Searching project logs, Matching invoice lines — not as another spinner that could mean anything. When the tool fails, keep prior text and say which step broke.
What we check before shipping a stream
Cancel mid-sentence and confirm the partial stays, the message is labeled incomplete, and the server stops generating. Drop the network after the first tokens and confirm the UI does not present the fragment as completed. Force a tool timeout and confirm the error is inline under the partial, with retry.
For structured outputs, kill the stream before the closing brace and confirm nothing invalid was written to storage. For citation-heavy answers like RuleCaddie, confirm a truncated reply never looks like a finished ruling with a quiet missing source.
Streaming is the right default for model latency. Completion is a separate event. If those two stay collapsed into one, the interface will keep training people to trust answers that never finished.
Questions
- Why not treat any closed stream as a completed answer?
- Sockets close for cancel, proxy drops, and tool failures as well as clean finishes. Without an explicit finish event and an incomplete state, truncated citations and half-written drafts look final and get trusted.
- Should Stop clear the partial response?
- No. Keep the text, mark it incomplete, re-enable input, and abort server-side generation. People often stop because they already have what they need; clearing forces a full regenerate.
- How do you preview structured JSON while it streams?
- Accumulate bytes and only promote fully closed fields, or show prose until the stream ends. Validate against the schema before committing to app state so a truncated object never becomes a saved draft.
Sources
- Vercel AI SDK — streaming and chat UI — useChat / streamText patterns BuilderHelp builds on.
- Vercel AI SDK — handling client disconnects — consumeStream and persistence when the tab drops mid-generation.
- Streaming chat interface practices — Abort, partial parse, and incomplete-message recovery patterns.