⚠️ 待更新·2026-08-29核验 · 更新时间待核验 · 本文信息可能已过期，请以官方文档为准
更新时间：2026-08-29 · 核验状态：待更新 · 官方溯源待补
背景
免费模型的稳定性天然低于付费:OpenRouter 免费模型经常 429、Groq 高峰时排队 5xx、Hugging Face 冷启动 30s。如果客户端直接裸调,每个调用点都要自己实现重试与降级,代码重复且容易写错。把这些策略集中到网关,客户端只关心业务结果。
核心策略
- 错误分类:网关把上游错误分成四类——
retryable(429、503、timeout)、non-retryable(400、401)、degradable(模型不支持工具)、fatal(5xx 持续)。每类有不同处理路径。
- 指数退避 + 抖动:重试间隔
base * 2^attempt + random(0, 1),避免同步重试打爆上游。基础 200ms,最多 4 次,总时长不超过 8s。
- 跨 Provider 降级:同模型在多家 Provider 都有副本(如
llama-3.3-70b 在 Groq 与 OpenRouter 都有)。第一次失败直接换 Provider,不要在同一 Provider 上死磕。
- 熔断器:同一 Provider 5 分钟内失败率 >50% 自动熔断 60s,期间流量绕过它。
- 降级链:用户请求
gpt-4o → 免费等价物 deepseek-chat:free → 更便宜的 llama-3.1-8b:free → 拒绝。每跳都记录在响应 header 里,客户端可观测。
代码示例
import asyncio, random, time
async def call_with_fallback(call_fn, providers, max_attempts=4):
for attempt in range(max_attempts):
provider = providers[attempt % len(providers)]
try:
return await call_fn(provider)
except (TimeoutError, RateLimitError) as e:
# exponential backoff + jitter
delay = 0.2 * (2 ** attempt) + random.random()
await asyncio.sleep(delay)
except (InternalError, ServiceUnavailable):
# immediately switch provider, no retry on same one
continue
# all attempts failed: degrade to cheaper model or raise
raise FallbackExhausted("tried: " + ",".join(providers))
部分重试语义
对于非流式的"多轮 tool 调用"场景,重试语义复杂:第一轮调用成功调了 tool,第二轮失败。如果整体重试,会触发 tool 的副作用两次。正确做法是幂等性 token:客户端为整条会话生成唯一 ID,网关在重试时把"已成功调用的 tool_call_id"作为透传头给上游,让上游跳过已执行的部分。没有幂等性 token 的 tool 调用,只允许 retry 在第一轮,后续失败必须整条失败。
最佳实践
- 重试要幂等:重试只对幂等请求(纯生成)安全;带
tool_calls 副作用的重试要更保守。
- 绝不重试流式:流式响应一旦开始吐 token 就不能再重试,首 chunk 之前失败才允许重试。
- 预算上限:为重试设最大消耗 token 数,避免一次失败连环触发 4 次重试把额度翻倍烧掉。
- 可观测:每次降级都打 metrics 标签
fallback_chain,运维能看出哪条链路最常触发。
- 断路器隔离:每家 Provider 独立断路器,避免一家连累另一家。
免费模型不是"能不能用"的问题,是"出问题时谁来兜"的问题,答案永远是网关。
⚠️ Pending Update · 2026-08-29 Verification · Content may be outdated, please refer to official docs
Updated: 2026-08-29 · Status: Pending Verification
Background
Free models are inherently less stable than paid: OpenRouter free models frequently 429, Groq returns 5xx during peak, Hugging Face cold starts can take 30s. If clients call them directly, every call site ends up implementing its own retry and fallback, which is both repetitive and error-prone. Concentrating these strategies in the gateway lets clients care only about the business result.
Core Strategies
- Error taxonomy: the gateway sorts upstream errors into four buckets —
retryable (429, 503, timeout), non-retryable (400, 401), degradable (model lacks tool support), fatal (sustained 5xx). Each bucket follows a different path.
- Exponential backoff with jitter: retry delay =
base * 2^attempt + random(0,1) to avoid synchronized retries hammering the upstream. Base 200ms, max 4 attempts, total under 8s.
- Cross-provider fallback: the same model often has replicas across providers (e.g.
llama-3.3-70b on both Groq and OpenRouter). On first failure, switch providers instead of beating the same one.
- Circuit breaker: if a provider's failure rate crosses 50% in a 5-minute window, break for 60s and route around it.
- Degradation chain: user requests
gpt-4o → free equivalent deepseek-chat:free → cheaper llama-3.1-8b:free → reject. Each hop is recorded in response headers so clients can observe it.
Code Example
import asyncio, random
async def call_with_fallback(call_fn, providers, max_attempts=4):
for attempt in range(max_attempts):
provider = providers[attempt % len(providers)]
try:
return await call_fn(provider)
except (TimeoutError, RateLimitError):
# exponential backoff + jitter
delay = 0.2 * (2 ** attempt) + random.random()
await asyncio.sleep(delay)
except (InternalError, ServiceUnavailable):
# immediately switch provider, no retry on same one
continue
# all attempts failed: degrade to cheaper model or raise
raise FallbackExhausted("tried: " + ",".join(providers))
Partial Retry Semantics
For non-streaming multi-turn tool-call scenarios, retry semantics get tricky: round one succeeded and called a tool, round two failed. If you retry the whole thing, the tool's side effect fires twice. The right approach is an idempotency token: the client generates a unique ID for the whole session, and the gateway passes "already-succeeded tool_call_ids" as a passthrough header so the upstream skips executed parts. Tool calls without an idempotency token allow retries only on the first round; later failures must fail the whole chain.
Best Practices
- Idempotency required: retries are safe only for idempotent requests (pure generation); retries on
tool_calls with side effects must be more conservative.
- Never retry streaming: once a stream has emitted tokens it cannot be retried — only pre-first-chunk failures may retry.
- Retry budget: cap maximum tokens spent on retries, so a single failure doesn't trigger four retries that double your burn.
- Observability: tag every fallback with a
fallback_chain metric label so ops can spot the most-traveled paths.
- Circuit-breaker isolation: each provider gets its own breaker so one's outage never takes down another.
The question with free models is never "can it be used" but "who catches it when it breaks," and the answer is always the gateway.