Building streaming LLM features that feel fast: the UX and engineering patterns

The biggest UX mistake in LLM-powered features is treating the model like a fast API. It isn't. A typical generation for a medium-length response runs 3–8 seconds to completion - if you wait for the full response before rendering anything, the user watches a spinner for most of that time and attributes the slowness to your product, not the model. Streaming is the table-stakes fix, but getting it right in a React app involves more than calling `.stream()` on an SDK. Here's the full pattern stack we've landed on.

The streaming primitive: Server-Sent Events

The transport layer for streaming LLM responses is Server-Sent Events (SSE). Unlike WebSockets, SSE is unidirectional (server to client), works over plain HTTP, respects standard load balancers, and doesn't require a handshake. Every major LLM API - Anthropic, OpenAI, Google - uses SSE or a compatible chunked transfer encoding. On the server, you open the response with `Content-Type: text/event-stream`, write each token as a `data:` line, and close with `data: [DONE]`.

// Server-side: TanStack Start server function streaming response
export const generateSummary = createServerFn()
  .validator(z.object({ content: z.string() }))
  .handler(async function* ({ data }) {
    const stream = await anthropic.messages.stream({
      model: 'claude-sonnet-4-6',
      max_tokens: 1024,
      messages: [{ role: 'user', content: `Summarize: ${data.content}` }],
    });
    for await (const chunk of stream) {
      if (chunk.type === 'content_block_delta') {
        yield chunk.delta.text;
      }
    }
  });

Rendering the stream in React

On the client, the pattern is a `useState` holding the accumulated text, with a `useEffect` that subscribes to the stream and appends each chunk. For React 19, `use()` with a streaming promise handles the Suspense integration. The key rendering detail: render each token as it arrives rather than batching them - the psychological benefit of visible incremental progress is the entire point of streaming from a UX standpoint.

function StreamingResponse({ prompt }: { prompt: string }) {
  const [text, setText] = useState('');
  const [done, setDone] = useState(false);

  useEffect(() => {
    setText('');
    setDone(false);
    // subscribe to the SSE stream and accumulate tokens
    const ctrl = new AbortController();
    streamResponse(prompt, ctrl.signal, (chunk) => {
      setText((prev) => prev + chunk);
    }).then(() => setDone(true));
    return () => ctrl.abort();
  }, [prompt]);

  return (
    <div>
      <Markdown>{text}</Markdown>
      {!done && <span className="animate-pulse">▋</span>}
    </div>
  );
}

Optimistic messages and rollback

In a chat interface, show the user's message immediately in the UI before the server confirms receipt. If the network request fails, remove the optimistic message and restore the input - TanStack Query's `onMutate` / `onError` pattern handles this cleanly. The perceived latency reduction from optimistic messages is significant: the user sees their input appear instantly, which separates the 'did it register my message?' anxiety from the model's response latency.

Graceful degradation

Model APIs go slow, hit rate limits, and return errors. Three fallback tiers we implement for every streaming feature: a 'still thinking' extended-wait state with an escape hatch after 10 seconds (not just a spinner - actual text telling the user what's happening), a cached previous response for users who hit the same prompt twice (deduplicate by hashed prompt, cache for 5 minutes), and a graceful error state with a retry button that doesn't lose the user's input. Teams building AI features almost never implement all three; teams operating AI features at scale always wish they had.

Abort on unmount and navigation

Always abort the stream when the component unmounts or the user navigates away. An un-aborted SSE connection keeps the model generating tokens, consuming API credits, and holding a server connection for a response nobody is reading. The `AbortController` pattern handles this: pass the `signal` to the fetch request and call `abort()` in the cleanup function of the `useEffect`. This is the most commonly skipped piece of the streaming implementation and the one that causes the most surprising billing surprises at scale.

Written by Appesto Engineering.