⚠️ 待更新·2026-08-29核验 · 更新时间待核验 · 本文信息可能已过期，请以官方文档为准
更新时间：2026-08-29 · 核验状态：待更新 · 官方溯源待补
背景
"想省钱"是模糊目标,网关需要把它拆成可执行的路由规则。核心思路:免费 Provider 是首选,付费 Provider 是兜底,缓存是放大器。每一层都有明确预算与触发条件,组合起来才能把成本压到最低。
路由规则集
按优先级从高到低执行,命中即停:
- L0 缓存命中:精确或语义缓存直接返回,成本 0。典型可拦 20-40% 流量。
- L1 免费层首选:在"今日剩余免费额度 > 阈值"的 Provider 里,按延迟+剩余额度排序。
- L2 免费降级:首选 Provider 失败或额度耗尽,切到次优免费 Provider。
- L3 付费兜底:所有免费 Provider 都不可用时,用付费 Provider,但限制每天总量。
- L4 模型降级:付费预算将尽时,把请求从
gpt-4o 降级到 gpt-4o-mini 或 llama-3.1-8b。
- L5 拒绝:所有预算耗尽,返回友好提示而非裸 503。
成本看板
- 每日成本曲线:观察付费层占比,目标 <10%。
- 缓存命中率:目标 25%+,低于则优化 key 设计。
- 降级触发次数:频繁触发说明免费层配额算小了,要扩 Provider 池。
代码示例
class FreeFirstRouter:
def __init__(self, cache, providers, daily_budget_cents=100):
self.cache = cache
self.providers = providers # sorted by free quota remaining
self.paid_used_cents = 0
self.daily_budget = daily_budget_cents
async def route(self, req):
# L0 cache
if cached := self.cache.get(req):
return cached, "L0-cache"
# L1/L2 free-first
for p in self.providers:
if p.free_remaining > 0:
try:
return await p.call(req), f"L1-{p.name}"
except (RateLimit, InternalError):
continue
# L3 paid fallback within budget
if self.paid_used_cents < self.daily_budget:
resp = await paid_provider.call(req)
self.paid_used_cents += resp.cost_cents
return resp, "L3-paid"
# L4 degrade
req["model"] = "llama-3.1-8b"
return await self.providers[0].call(req), "L4-degraded"
Provider 池容量规划
"免费优先"策略的成败取决于 Provider 池容量。太少(1-2 家)任何一家限流就降级到付费,省钱效果差;太多(10 家)运维成本暴涨,每家都要监控。推荐 3-5 家免费 Provider + 1-2 家付费兜底。每家 Provider 配额要满足"覆盖单日峰值的 30%",这样任意 1 家宕机,其他 2-3 家能凑出余量。每周复盘"哪家常宕、哪家常超量",动态调整池成员。
最佳实践
- 预算护栏:软上限(降级)+ 硬上限(拒绝),软 < 硬。
- 免费池要分散:至少挂 3 家免费 Provider,避免单家限流连累全局。
- 缓存预热:对已知高频问题,每天凌晨批量预生成缓存。
- 复盘:每周复盘降级链命中率,优化 Provider 排序与预算分配。
- 跨区设计:免费 Provider 跨多个地理区域选,某区域故障仍有备份。
把"免费优先"做成代码,而不是写成 OKR 的口号。
⚠️ Pending Update · 2026-08-29 Verification · Content may be outdated, please refer to official docs
Updated: 2026-08-29 · Status: Pending Verification
Background
"Save money" is a vague goal; the gateway needs to decompose it into executable routing rules. The core idea: free providers are the first choice, paid providers are the fallback, and the cache is the amplifier. Each layer has explicit budgets and triggers; only when composed do they squeeze cost to the minimum.
Routing Rule Set
Execute in priority order, top to bottom; stop on the first hit:
- L0 cache hit: exact or semantic cache returns directly at zero cost. Typically catches 20-40% of traffic.
- L1 free-tier first: among providers whose daily free quota remaining crosses a threshold, sort by latency + remaining quota.
- L2 free degradation: if the first choice fails or quota runs out, switch to the next free provider.
- L3 paid fallback: when no free provider is available, use a paid one, but cap the daily total.
- L4 model downgrade: as the paid budget approaches exhaustion, downgrade the request from
gpt-4o to gpt-4o-mini or llama-3.1-8b.
- L5 reject: when every budget is exhausted, return a friendly notice instead of a bare 503.
Cost Dashboard
- Daily cost curve: watch the paid-tier ratio; target <10%.
- Cache hit rate: target 25%+; below that, redesign the key.
- Downgrade trigger count: frequent triggers mean the free pool is too small — expand it.
Code Example
class FreeFirstRouter:
def __init__(self, cache, providers, daily_budget_cents=100):
self.cache = cache
self.providers = providers # sorted by free quota remaining
self.paid_used_cents = 0
self.daily_budget = daily_budget_cents
async def route(self, req):
# L0 cache
if cached := self.cache.get(req):
return cached, "L0-cache"
# L1/L2 free-first
for p in self.providers:
if p.free_remaining > 0:
try:
return await p.call(req), f"L1-{p.name}"
except (RateLimit, InternalError):
continue
# L3 paid fallback within budget
if self.paid_used_cents < self.daily_budget:
resp = await paid_provider.call(req)
self.paid_used_cents += resp.cost_cents
return resp, "L3-paid"
# L4 degrade
req["model"] = "llama-3.1-8b"
return await self.providers[0].call(req), "L4-degraded"
Provider Pool Capacity Planning
The free-first strategy's success depends on the provider pool size. Too small (1-2 providers) and any rate limit cascades to paid — savings vanish; too large (10 providers) and operational cost explodes, every provider needs monitoring. Recommended: 3-5 free providers + 1-2 paid fallbacks. Each provider's quota should cover 30% of daily peak, so any one going down leaves the other 2-3 with enough headroom. Weekly review of "which provider is always down, which is always over quota" tunes pool membership dynamically.
Best Practices
- Budget guardrails: a soft cap (downgrade) + a hard cap (reject); soft sits below hard.
- Diversify the free pool: register at least three free providers so a single rate limit does not cascade.
- Cache warming: for known high-frequency questions, batch-generate the cache in a nightly job.
- Weekly review: audit downgrade-chain hit rates weekly and tune provider ordering and budget allocation.
- Cross-region design: pick free providers across multiple regions so a regional outage still leaves backups.
Make "free first" into code, not an OKR slogan.