Fallback and Retry Patterns

Production Reality: Things Fail

In production, LLM APIs fail more often than you'd think:

• Rate limits: Hit token or request limits
• Timeouts: Model takes too long to respond
• Server errors: 500, 502, 503 errors
• Content filters: Request blocked by safety filters
• Outages: Full API outages (rare but happen)

Retry Strategy: Exponential Backoff

Don't hammer the API when it's struggling. Use exponential backoff:

Fallback Chain Pattern

Error Classification

Error Type	Should Retry?	Should Fallback?
Rate limit (429)	Yes, with backoff	Yes, immediately
Timeout	Yes, once	Yes
Server error (5xx)	Yes, with backoff	After retries
Bad request (400)	No	No (fix the request)
Auth error (401/403)	No	No (fix credentials)