⚠️ 待更新·2026-08-29核验 · 更新时间待核验 · 本文信息可能已过期，请以官方文档为准
更新时间：2026-08-29 · 核验状态：待更新 · 官方溯源待补
背景
不同 Provider 的计费单位五花八门:OpenAI 按 token、Gemini 按字符、Replicate 按秒、Cohere Trial 按月请求数。免费层又叠加 RPM、RPD、TPM、IPM 多维限制。把它们都算成"用户花了多少"需要网关在每次响应回来后做一次归一化计量,再回写到统一的额度账本。
核心设计
- 统一计量单位:把所有上游的成本折算成"标准 token"(normalized token),1 美元等价 = 1 单位 credit。免费模型记 0 成本但仍记调用次数。
- 三层额度账本:
user_quota(用户月总额度)→ client_quota(每个 client_id 日额度)→ provider_quota(每 Provider 的免费额度池)。三层独立扣减,先紧的先触发降级。
- 免费优先策略:路由时按"剩余免费额度倒序"选 Provider,免费池见底才切到付费池,实现"把每一分钱都省下来"。
- 超额降级:触发某层额度上限时,网关自动把请求降级到更便宜的模型(如 70B→8B),而不是直接拒绝。
- 对账:每天凌晨拉每家 Provider 的官方账单 API,与网关侧计量结果对账,差异 >5% 触发告警。
代码示例
class QuotaLedger:
def __init__(self, redis):
self.r = redis
def consume(self, client_id, provider, tokens, is_free):
# decrement three layers atomically
pipe = self.r.pipeline()
pipe.hincrby(f"u:{client_id}", "spent", tokens)
pipe.hincrby(f"c:{client_id}:day", "spent", tokens)
if is_free:
pipe.hincrby(f"p:{provider}:free", "spent", tokens)
pipe.execute()
def route(self, client_id, candidates):
# free first, sort by remaining free quota desc
free = [(p, self.r.get(f"p:{p}:free:remain") or 0) for p in candidates]
free.sort(key=lambda x: -int(x[1]))
for p, _ in free:
if self._can_use(p, client_id):
return p, True
return candidates[0], False # fall back to paid
实时计量 vs 批量计量
计费有两种粒度:实时计量(每响应回写)与批量计量(每 5 分钟 flush)。实时计量让路由能立刻看见"这家额度快没了",但每响应一次 Redis 写,有性能成本;批量计量延迟低,但路由层 5 分钟内看不到刚烧完的额度。折中:用 in-memory 计数器累加,每 50 次或 5 秒 flush 一次 Redis,平衡延迟与写入压力。
最佳实践
- 预算护栏:设硬上限(超额拒绝)与软上限(超额降级),两者数值不同,软 < 硬。
- 冷启动数据:新用户先给 1000 标准 token 试用,用完触发付费引导,而不是直接拒绝。
- 预扣减:流式响应边吐边扣,不要等结束再扣,避免半路超支。
- 对账自动化:每日凌晨拉 Provider 账单 API 与网关计量对账,差异 >5% 告警。
计费不是后台报表,而是路由的实时输入。
⚠️ Pending Update · 2026-08-29 Verification · Content may be outdated, please refer to official docs
Updated: 2026-08-29 · Status: Pending Verification
Background
Provider billing units are wildly inconsistent: OpenAI bills by token, Gemini by character, Replicate by second, and Cohere Trial by monthly request count. Free tiers pile on multi-dimensional limits — RPM, RPD, TPM, IPM. To produce a single "how much did the user spend" view, the gateway must normalize metering after each upstream response and write it back to a unified ledger.
Core Design
- Unified unit: fold every upstream cost into "normalized tokens," where 1 credit = $1 equivalent. Free models record zero cost but still count call volume.
- Three-layer ledger:
user_quota (monthly total per user) → client_quota (daily per client_id) → provider_quota (free pool per provider). Layers decrement independently; the tightest layer triggers degradation first.
- Free-first policy: routing sorts providers by remaining free quota descending; only when the free pool is empty does it switch to paid — this is "save every cent" as an executable policy.
- Overage degradation: on hitting a layer cap, the gateway degrades the request to a cheaper model (e.g. 70B → 8B) instead of rejecting it.
- Reconciliation: a nightly job pulls each provider's official billing API and compares to gateway metering; >5% deviation triggers an alert.
Code Example
class QuotaLedger:
def __init__(self, redis):
self.r = redis
def consume(self, client_id, provider, tokens, is_free):
# decrement three layers atomically
pipe = self.r.pipeline()
pipe.hincrby(f"u:{client_id}", "spent", tokens)
pipe.hincrby(f"c:{client_id}:day", "spent", tokens)
if is_free:
pipe.hincrby(f"p:{provider}:free", "spent", tokens)
pipe.execute()
def route(self, client_id, candidates):
# free first, sort by remaining free quota desc
free = [(p, self.r.get(f"p:{p}:free:remain") or 0) for p in candidates]
free.sort(key=lambda x: -int(x[1]))
for p, _ in free:
if self._can_use(p, client_id):
return p, True
return candidates[0], False # fall back to paid
Real-Time vs Batch Metering
Billing has two granularities: real-time metering (write back per response) and batch metering (flush every 5 minutes). Real-time metering lets routing see "this provider's quota is nearly exhausted" immediately, but a Redis write per response has performance cost; batch metering has lower write pressure but routing cannot see the just-burned quota for 5 minutes. Compromise: accumulate in an in-memory counter, flush to Redis every 50 calls or 5 seconds, balancing latency with write pressure.
Best Practices
- Budget guardrails: define a soft cap (degrade on overage) and a hard cap (reject on overage); the soft cap sits below the hard cap.
- Cold-start credits: give new users 1000 normalized tokens up front, then prompt for a paid plan when exhausted — never just reject.
- Pre-debit streaming: deduct incrementally as a streaming response emits tokens, not at the end, to prevent mid-stream overruns.
- Automated reconciliation: a nightly job pulls each provider's billing API and reconciles against gateway metering; >5% deviation alerts.
Billing is not a back-office report — it is a real-time input to routing.