多模型负载均衡

⚠️ 待更新·2026-08-29核验 · 更新时间待核验 · 本文信息可能已过期，请以官方文档为准更新时间：2026-08-29 · 核验状态：待更新 · 官方溯源待补

背景

单 Provider 撑不住高并发:Groq 免费 RPM=30、OpenRouter 免费 20 RPM。但如果你同时挂 5 家,理论总和 100 RPM,足够小产品用。负载均衡层把这 5 家当成一个"逻辑 Provider"对外暴露,内部按策略分摊。

负载均衡策略

加权轮询(Weighted Round-Robin):每家按"剩余免费额度 / 平均延迟"算权重,额度多且快的家权重大。简单稳定,默认选它。
最少在途(Least Connections):跟踪每家当前 in-flight 请求数,优先发给最闲的。适合长连接 / 流式场景。
延迟感知(P2C + EWMA):Pick-of-2 挑两个候选,选 EWMA 延迟更低的。来自 Nginx 实践,对延迟抖动敏感。
配额感知:跳过当日额度已耗尽的 Provider,不浪费重试次数。
粘性(Sticky):同一会话尽量路由到同一 Provider,避免不同家的 token 计数差异影响上下文窗口预算。

代码示例

import time, random
from collections import defaultdict

class LoadBalancer:
    def __init__(self, providers):
        # provider -> {"weight": int, "ewma_latency": float, "inflight": int}
        self.p = {k: {"weight": v, "ewma_latency": 100, "inflight": 0}
                  for k, v in providers.items()}
        self.last_pick = None

    def pick(self) -> str:
        # Pick-of-2 by latency, weighted fallback
        candidates = random.sample(list(self.p.keys()), 2)
        # filter out quota-exhausted ones (caller should mark)
        candidates = [c for c in candidates if self.p[c]["inflight"] < 5]
        if not candidates:
            return min(self.p, key=lambda k: self.p[k]["inflight"])
        return min(candidates, key=lambda k: self.p[k]["ewma_latency"])

    def observe(self, provider, latency, ok):
        a = 0.3  # EWMA smoothing
        self.p[provider]["ewma_latency"] = (
            (1-a) * self.p[provider]["ewma_latency"] + a * latency)
        if not ok:
            self.p[provider]["weight"] = max(1, self.p[provider]["weight"] // 2)

健康检查节奏

后台健康检查节奏很关键:太频繁(每秒)浪费 Provider 配额,太稀疏(每分钟)感知慢。推荐 5 秒一次轻量探针(打 /models),15 秒一次重度探针(发一个最便宜的真实 prompt)。轻度探针测连通性,重度探针测推理可用性,两者结合才能区分"网络通但服务挂"与"完全离线"。重度探针的成本计入 Provider 配额,所以只跑在没流量的空闲 Provider 上。

权衡与最佳实践

不要纯随机:随机分布会让单家偶发尖峰,加权 + P2C 才稳。
健康检查异步:后台 5s 一次 ping /models,不要让真实请求承担探测成本。
预热新 Provider:新加的 Provider 起始权重给 5% 而不是 100%,逐步爬升避免冷启动雪崩。
请求粘性:同一会话尽量路由到同一 Provider,避免上下文窗口计数差异。
池上限保护:每家 Provider 设最大并发上限,防止某家被过度打爆。

负载均衡不是"公平",而是"让每家 Provider 在它的舒适区工作"。

⚠️ Pending Update · 2026-08-29 Verification · Content may be outdated, please refer to official docs Updated: 2026-08-29 · Status: Pending Verification

Background

A single provider cannot sustain high concurrency: Groq free tier RPM=30, OpenRouter free 20 RPM. But if you register five providers simultaneously, the theoretical total of 100 RPM is plenty for a small product. The load balancer exposes these five as one "logical provider," distributing internal traffic by policy.

Load-Balancing Strategies

Weighted Round-Robin: each provider's weight = remaining free quota / average latency; the more generous and faster, the higher the weight. Simple and stable — the default.
Least Connections: tracks each provider's in-flight requests and prefers the idlest. Suits long-lived / streaming scenarios.
Latency-aware (P2C + EWMA): Pick-of-2 candidates, choose the one with lower EWMA latency. Originated in Nginx, sensitive to latency jitter.
Quota-aware: skip providers whose daily quota is exhausted, so retries are not wasted.
Sticky: route the same session to the same provider to avoid context-window drift caused by differing token counters.

Code Example

import time, random
from collections import defaultdict

class LoadBalancer:
    def __init__(self, providers):
        # provider -> {"weight": int, "ewma_latency": float, "inflight": int}
        self.p = {k: {"weight": v, "ewma_latency": 100, "inflight": 0}
                  for k, v in providers.items()}

    def pick(self) -> str:
        # Pick-of-2 by latency, weighted fallback
        candidates = random.sample(list(self.p.keys()), 2)
        candidates = [c for c in candidates if self.p[c]["inflight"] < 5]
        if not candidates:
            return min(self.p, key=lambda k: self.p[k]["inflight"])
        return min(candidates, key=lambda k: self.p[k]["ewma_latency"])

    def observe(self, provider, latency, ok):
        a = 0.3  # EWMA smoothing
        self.p[provider]["ewma_latency"] = (
            (1-a) * self.p[provider]["ewma_latency"] + a * latency)
        if not ok:
            self.p[provider]["weight"] = max(1, self.p[provider]["weight"] // 2)

Health Check Cadence

Background health check cadence is critical: too frequent (per second) burns provider quota, too sparse (per minute) lags detection. Recommended: 5-second light probes (hit /models), 15-second heavy probes (send one cheapest real prompt). Light probes test connectivity, heavy probes test inference availability — together they distinguish "network up but service down" from "fully offline." Heavy probes cost provider quota, so run them only on idle providers with no traffic.

Trade-offs and Best Practices

Avoid pure random: random distribution causes occasional single-provider spikes; weighted + P2C is more stable.
Async health checks: ping /models every 5s in the background — never make real requests bear the probing cost.
Warm up new providers: a newly added provider starts at 5% weight, not 100%, ramping up to avoid cold-start stampedes.
Sticky sessions: route the same session to the same provider to avoid context-window drift.
Pool cap protection: set a max-concurrency per provider so one cannot be over-stamped.

Load balancing is not "fairness" — it is "keeping each provider inside its comfort zone."