The LLM Retry Loop That Looks Like Progress and Does Nothing
Why Blind Retries Turn LLM Failures into Silent Data Poison and How to Implement Smart Backoff Logic
What Is An LLM Retry Loop Failure?
An LLM retry loop failure occurs when a system repeatedly retries a structural error, such as a prompt-induced hallucination or a schema mismatch, rather than a transient network error. This "looks like progress" in logs but results in wasted API costs and data poisoning, where the model eventually produces valid-looking but semantically incorrect output.
The most dangerous failure mode in LLM production systems is not the one that crashes. It is the one that keeps running, logs success, and returns garbage.
Retry logic is one of the first things engineers add to LLM pipelines. Models fail transiently. Rate limits happen. Network timeouts happen. A retry wrapper around your API call is reasonable defensive engineering. The problem is that LLM failures are not all transient. Some failures are structural. When you apply a retry loop to a structural failure, you do not get recovery. You get a loop that runs to its maximum count, burns tokens, burns time, and either raises a final exception or returns a degraded output that passes downstream validation and poisons everything that depends on it.
This is the retry loop failure mode. It is extremely common and almost never discussed.
Why Retry Logic Makes Sense for LLMs in Theory
The reasoning behind retry wrappers is sound. LLM API calls fail for reasons unrelated to your prompt or your code. Rate limiting returns a 429. A cold model instance returns a 503. A network blip drops a connection mid-stream. These failures are transient, meaning the exact same call, retried after a short wait, will typically succeed.
The standard pattern handles these cases well:
import time
import random
import logging
from openai import OpenAI, RateLimitError, APIConnectionError, APIStatusError
from pydantic import ValidationError
# Setup logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
client = OpenAI()
# Errors worth retrying: transient infrastructure issues
RETRYABLE_ERRORS = (RateLimitError, APIConnectionError)
# Errors not worth retrying: structural problems (400: Bad Request, 401: Auth, etc.)
NON_RETRYABLE_STATUS_CODES = {400, 401, 403, 404}
def call_with_smart_retry(prompt: str, max_retries: int = 3):
"""
Executes an LLM call with exponential backoff and jitter.
Distinguishes between transient network/rate errors and structural API errors.
"""
for attempt in range(max_retries):
logger.info(f"Attempt {attempt + 1}/{max_retries} - prompt preview: {prompt[:80]}...")
try:
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
timeout=60.0 # Prevent hanging on slow connections
)
return {
"content": response.choices[0].message.content,
"stop_reason": response.choices[0].finish_reason,
"usage": response.usage.model_dump() if response.usage else None
}
except RETRYABLE_ERRORS as e:
logger.warning(f"Retryable error on attempt {attempt + 1}: {str(e)}")
if attempt == max_retries - 1:
raise
# Exponential backoff: 2, 4, 8 seconds + small jitter
wait = (2 ** (attempt + 1)) + random.uniform(0, 0.3)
time.sleep(wait)
except APIStatusError as e:
# Safely extract message from the OpenAI error object
error_message = getattr(e, 'message', str(e))
logger.warning(f"API status error on attempt {attempt + 1}: {e.status_code} - {error_message}")
if e.status_code in NON_RETRYABLE_STATUS_CODES:
# Structural failure: stop immediately to save tokens/time
raise RuntimeError(
f"Non-retryable API error {e.status_code}: {error_message}"
) from e
# For 500-level errors or unknown codes, retry cautiously
if attempt == max_retries - 1:
raise
wait = (2 ** (attempt + 1)) + random.uniform(0, 0.3)
time.sleep(wait)
def validate_with_retry(raw_output: str, model_class, max_retries: int = 2):
"""
Validates LLM output against a Pydantic schema.
Only retries if the error looks like a recoverable formatting glitch.
"""
for attempt in range(max_retries):
try:
return model_class.model_validate_json(raw_output)
except ValidationError as e:
logger.error(f"Validation attempt {attempt + 1} failed.")
if attempt == max_retries - 1:
raise RuntimeError(
f"Validation failed after {max_retries} attempts. "
f"Last output: {raw_output[:500]}"
) from e
# Heuristic: Retry only if the model missed a field or added an extra one.
# If the content is fundamentally wrong, a retry usually won't help.
error_str = str(e).lower()
if "missing" in error_str or "extra" in error_str:
logger.info("Detected recoverable formatting error. Retrying...")
continue
else:
raise # Content validation failure: not retryableThis works. For transient failures, it works well. The problem begins when this same pattern is applied to failures that are not transient.
The Four Failure Types That Break Retry Logic
Prompt-induced failures. If your prompt reliably causes the model to produce output that fails your validation logic, retrying produces the same failed output every time. The model is not broken. Your prompt is producing a consistent response that your validation rejects. Three retries later you have burned three times the tokens and you are exactly where you started.
Schema mismatch failures. If you are using structured output and your Pydantic schema does not match what the model can reliably produce, retry logic loops indefinitely. Instructor’s built-in retry with repair is designed for this, sending the validation error back to the model and asking it to fix the output. If the schema mismatch is fundamental rather than incidental, the repair prompt does not help either.
Context window failures. If your input exceeds the model’s context window, the API returns an error. Retrying the same oversized input returns the same error. This is a structural failure that requires reducing the input size, not retrying.
Model capability failures. Some tasks are outside a given model’s reliable capability. If you are asking a smaller model to perform consistent multi-step reasoning that it cannot reliably execute, retrying does not change the model’s capabilities. You get the same unreliable output on each attempt.
The Silent Success Problem
The most damaging variant of the retry loop failure is not the one that exhausts retries and raises an exception. It is the one that returns on the second or third attempt with output that passes validation but is subtly wrong.
Consider a pipeline that extracts structured data from documents. On the first attempt, the model returns malformed JSON. The retry wrapper catches the exception and retries. On the second attempt, the model returns valid JSON that passes Pydantic validation but contains hallucinated values for fields where the source document was ambiguous. The retry logic logs a success. The hallucinated data flows downstream.
This failure mode is invisible in your error logs. Your retry metrics show a healthy success rate. Your downstream systems are operating on corrupted data. You will not discover it until something that depends on that data produces a wrong result and someone traces it back.
How to Build Retry Logic That Distinguishes Failure Types
The fix is to categorize failures before deciding whether to retry.
import time
from openai import OpenAI, RateLimitError, APIConnectionError, APIStatusError
client = OpenAI()
# Errors worth retrying: transient infrastructure issues
RETRYABLE_ERRORS = (RateLimitError, APIConnectionError)
# Errors not worth retrying: structural problems
NON_RETRYABLE_STATUS_CODES = {400, 401, 403, 404}
def call_with_smart_retry(prompt: str, max_retries: int = 3):
for attempt in range(max_retries):
try:
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.content
except RETRYABLE_ERRORS as e:
if attempt == max_retries - 1:
raise
wait = 2.0 * (attempt + 1)
time.sleep(wait)
except APIStatusError as e:
if e.status_code in NON_RETRYABLE_STATUS_CODES:
# Structural failure: do not retry
raise RuntimeError(
f"Non-retryable API error {e.status_code}: {e.message}"
) from e
# Unknown status code: retry cautiously
if attempt == max_retries - 1:
raise
time.sleep(2.0)For validation failures, add explicit categorization:
def validate_with_retry(raw_output: str, model_class, max_retries: int = 2):
for attempt in range(max_retries):
try:
return model_class.model_validate_json(raw_output)
except ValidationError as e:
if attempt == max_retries - 1:
# Log the validation failure with the actual output for diagnosis
raise RuntimeError(
f"Validation failed after {max_retries} attempts. "
f"Last output: {raw_output[:500]}"
) from e
# Only retry validation if the failure looks like a formatting issue
# not a content issue
if "missing" in str(e).lower() or "extra" in str(e).lower():
continue
else:
raise # Content validation failure: not retryableThe Logging You Need to Actually Diagnose This
Retry loops that fail silently are invisible without the right observability. At minimum, log the following for every retry attempt:
The attempt number and the exception type
The raw model output before validation, truncated to a readable length
Whether the retry was triggered by an infrastructure error or a validation error
The final outcome: success, exhausted retries, or non-retryable failure
Without this logging, you will spend hours looking at a system that reports normal operation while producing wrong results.
If You Read This Far, My Weekly AI Newsletter Is Probably For You.
Every Wednesday I send Pithy Cyborg | AI News Made Simple → 3 elite AI stories plus one prompt, no advertisers, no sponsors, no outside funding. One person. 10 to 20 hours of research. Straight to your inbox.
Always free. No paywalls. If it matters to you, a paid subscription ($5/month or $40/year) is what keeps it independent.
Subscribe free → Join Pithy Cyborg | AI News Made Simple for free.
Upgrade to paid → Become a paid subscriber. Support independent AI journalism.
If you’re not ready to subscribe, following on social helps more than you might think.
✖️ X/Twitter | 🦋 Bluesky | 💼 LinkedIn | ❓ Quora | 👽 Reddit
Thanks for reading.
Cordially yours,
Mike D (aka MrComputerScience)
Pithy Cyborg | AI News Made Simple
PithyCyborg.Substack.com