Why Your Streaming LLM Endpoint Hangs Under Load | The 3 Silent Production Killers
From Worker Thread Exhaustion to Proxy Buffering | Fixing Connection Failures in Real-Time AI Apps
What Causes Streaming LLM Endpoints To Fail In Production?
Streaming LLM endpoints typically fail under load due to worker thread exhaustion, where long-lived connections block synchronous web servers, and proxy buffering, where intermediate servers (like Nginx) wait for the full response before forwarding tokens. Additionally, default load balancer timeouts often terminate long-running generations before they complete. The solution involves using async workers, disabling proxy buffering, and implementing application-level heartbeats.
Local development → works every time. Staging environment → works fine. First week in production with real users? Hangs intermittently, then consistently, then the on-call alert fires at 2am.
Streaming LLM endpoints have a failure mode under load that is almost invisible in local and staging environments and highly visible in production. It is not a bug in any single component. It is an interaction between the way streaming responses work, the way web servers handle long-lived connections, and the way load balancers and proxies manage connection state.
Understanding this failure requires understanding what streaming actually does to your infrastructure.
What Streaming Does to Your Web Server
A standard HTTP request-response cycle is brief. The client sends a request, the server processes it, the server sends a response, and the connection closes or is returned to a pool. The server resource is held for seconds or less.
A streaming LLM response is different. The connection stays open for the entire duration of token generation. For a response that takes 15 seconds to generate, the server is holding that connection open for 15 seconds. For a response that takes 45 seconds for a complex task, the connection is held for 45 seconds.
Under low load, this is manageable. Under production load, this creates a resource exhaustion problem that the server metrics will show clearly once you know what to look for but that does not appear at all in local testing where concurrency is low.
The Worker Thread Exhaustion Pattern
Most Python web frameworks, including Flask with Gunicorn and FastAPI with Uvicorn in default configurations, have a limited number of worker threads or processes available to handle requests. The exact number depends on your server configuration, typically 2 to 4 workers per CPU core as a starting point.
Each streaming LLM request holds a worker thread for its entire duration. If your server has 8 worker threads and 8 concurrent users each making a streaming request that takes 20 seconds, all 8 workers are occupied for 20 seconds. The ninth request queues. If the queue fills, requests start timing out or receiving connection refused errors.
This is a straightforward resource exhaustion problem. What makes it confusing is that it does not appear at low concurrency. With 2 or 3 concurrent users, your 8 workers handle the load comfortably. The first symptom appears when concurrency reaches the worker count, and by then you are in production.
The Load Balancer Timeout Problem
Load balancers and reverse proxies impose connection timeouts to protect against hung connections. A common default is 60 seconds. For most web applications, this is more than sufficient. For streaming LLM endpoints generating long responses, it is not.
If your load balancer has a 60-second timeout and your LLM generates a response that takes 75 seconds, the load balancer closes the connection at 60 seconds. The client receives an abrupt disconnection. The server may continue generating tokens into a closed connection, wasting compute and API tokens.
This failure does not appear locally because there is no load balancer in local development. It does not appear in simple staging environments that do not replicate production load balancer configuration. It appears in production on the first long-running request that exceeds the timeout threshold.
The Buffering Problem
Some reverse proxy configurations buffer response bodies before forwarding them to the client. Nginx, for example, has proxy_buffering enabled by default. When buffering is enabled, the proxy waits to accumulate the full response before forwarding it, which completely defeats the purpose of streaming and can cause timeout failures on long responses.
The symptom is that streaming works when you hit the application directly but tokens are not delivered progressively when going through the proxy. Users see a blank response for the full generation time, then the complete response appears at once, then it stops working entirely for longer responses because the buffer timeout fires.
The Complete Fix Stack
Each of these problems has a specific fix.
Worker exhaustion → Use async workers.
Replace synchronous Gunicorn workers with async workers using uvicorn or uvicorn workers with Gunicorn. Async workers handle I/O-bound operations like waiting for streaming API responses without blocking a thread.
# Instead of:
gunicorn app:app -w 4
# Use:
gunicorn app:app -w 4 -k uvicorn.workers.UvicornWorkerWith async workers, a single worker can handle many concurrent streaming connections because it yields the event loop while waiting for tokens rather than blocking a thread.
Load balancer timeouts → Increase and configure appropriately.
For AWS ALB:
Idle timeout → set to 300 seconds or higher depending on your maximum expected response time
Enable HTTP/2 for better connection handling
For Nginx:
proxy_read_timeout 300s;
proxy_send_timeout 300s;
keepalive_timeout 300s;Buffering → Disable it for streaming endpoints.
In Nginx, disable buffering for your LLM endpoint:
location /api/stream {
proxy_pass http://backend;
proxy_buffering off;
proxy_cache off;
proxy_set_header Connection '';
proxy_http_version 1.1;
chunked_transfer_encoding on;
}Application layer → Add heartbeat tokens for long responses.
For responses that may take more than 30 seconds, send periodic whitespace or comment tokens to keep the connection alive and prevent intermediate proxies from closing the connection due to inactivity:
async def stream_with_keepalive(prompt: str):
import asyncio
import anthropic
client = anthropic.AsyncAnthropic()
async with client.messages.stream(
model="claude-sonnet-4-20250514",
max_tokens=4096,
messages=[{"role": "user", "content": prompt}]
) as stream:
last_token_time = asyncio.get_event_loop().time()
async for text in stream.text_stream:
yield text
last_token_time = asyncio.get_event_loop().time()
# Send keepalive if no tokens in 15 seconds
current_time = asyncio.get_event_loop().time()
if current_time - last_token_time > 15:
yield " " # Whitespace keepaliveThe Load Test You Should Run Before Launch
Before deploying a streaming LLM endpoint to production, run a load test that simulates realistic concurrency with realistic response durations. Tools like k6 or Locust can simulate multiple concurrent users each holding a streaming connection open for 15 to 60 seconds.
The metric to watch is not response time on individual requests. It is success rate as concurrency increases. If success rate drops when concurrency reaches 8 or 10 simultaneous users, you have a worker exhaustion problem. If long responses fail but short ones succeed, you have a timeout problem. Both are fixable before users find them.
If You Read This Far, My Weekly AI Newsletter Is Probably For You.
Every Wednesday I send Pithy Cyborg | AI News Made Simple → 3 elite AI stories plus one prompt, no advertisers, no sponsors, no outside funding. One person. 10 to 20 hours of research. Straight to your inbox.
Always free. No paywalls. If it matters to you, a paid subscription ($5/month or $40/year) is what keeps it independent.
Subscribe free → Join Pithy Cyborg | AI News Made Simple for free.
Upgrade to paid → Become a paid subscriber. Support independent AI journalism.
If you’re not ready to subscribe, following on social helps more than you might think.
✖️ X/Twitter | 🦋 Bluesky | 💼 LinkedIn | ❓ Quora | 👽 Reddit
Thanks for reading.
Cordially yours,
Mike D (aka MrComputerScience)
Pithy Cyborg | AI News Made Simple
PithyCyborg.Substack.com