⚠️ 待更新·2026-08-29核验 · 更新时间待核验 · 本文信息可能已过期，请以官方文档为准
更新时间：2026-08-29 · 核验状态：待更新 · 官方溯源待补
背景
模型调用天然是黑盒:一次失败可能是上游 5xx、Key 失效、Prompt 超长、限流,也可能只是 DNS 抖动。没有统一可观测性,运维只能靠用户投诉定位。把日志/指标/trace 在网关层一次性采集,后续任何排障都从同一份数据出发。
核心三件套
- Metrics:每次请求记录
latency_p50/p95/p99、success_rate、tokens_in/out、cost_cents、fallback_count。按 client_id × provider × model 三维切分。
- Logs:结构化 JSON 日志,字段包含
request_id、client_id、provider、upstream_latency、error_class、fallback_chain。保留 7 天明细 + 90 天聚合。
- Traces:用 OpenTelemetry 给每条请求打一个 trace,贯穿"客户端 → 网关 → 上游 → 解析 → 返回"。一次失败能秒级定位卡在哪一段。
关键指标看板
- 健康四象限:延迟(上游 + 网关)、成功率、成本(日均/月累计)、配额剩余。每项设 SLO 与告警阈值。
- provider 热力图:每家 Provider 在不同时段的成功率分布,据此调整路由权重。
- prompt 长度直方图:长 prompt 会撑爆上下文,提前发现"用户输入暴增"。
代码示例
from opentelemetry import trace, metrics
tracer = trace.get_tracer("gateway")
meter = metrics.get_meter("gateway")
latency_hist = meter.create_histogram("gateway.latency", unit="ms")
token_counter = meter.create_counter("gateway.tokens")
async def call_upstream(req, provider):
with tracer.start_as_current_span(f"upstream.{provider}") as span:
span.set_attribute("model", req["model"])
t0 = time.time()
try:
resp = await http.post(...)
latency_hist.record((time.time()-t0)*1000, {
"provider": provider, "model": req["model"]})
token_counter.add(resp["usage"]["total_tokens"], {
"provider": provider, "dir": "total"})
return resp
except Exception as e:
span.record_exception(e)
span.set_status(trace.Status(trace.StatusCode.ERROR))
raise
成本 vs 可观测性平衡
全量采集 trace 会让存储爆炸,完全不采又失去排障能力。平衡点在采样分级:正常请求采 1%、错误请求 100% 采、慢请求(P99 阈值)100% 采。这样 99% 的请求都不入库,但所有"值得看"的请求都在。另外,日志保留分级:实时明细 7 天、聚合统计 90 天、长期归档 1 年冷存储,查询性能与存储成本双优。
最佳实践
- 采样策略:trace 全采会爆存储,按 1% 随机采样 + 100% 采错误请求。
- PII 脱敏:Prompt 里常含用户隐私,日志落地前要替换 email/手机号。
- SLO 燃烧率:用多窗口多燃烧率告警(5m × 1h + 1h × 6h),比静态阈值更早发现异常。
- 统一 ID 串联:同一请求 trace_id / span_id / log_id 三者一致,便于跨表查询。
可观测性不是"出事再查",而是"不出事也能看见"。
⚠️ Pending Update · 2026-08-29 Verification · Content may be outdated, please refer to official docs
Updated: 2026-08-29 · Status: Pending Verification
Background
Model calls are inherently a black box: a single failure could be upstream 5xx, an expired key, an over-long prompt, a rate limit, or just DNS jitter. Without unified observability, ops relies on user complaints for diagnosis. Capturing logs, metrics, and traces at the gateway layer means every investigation starts from the same data set.
The Observability Triad
- Metrics: each request records
latency_p50/p95/p99, success_rate, tokens_in/out, cost_cents, fallback_count, sliced across client_id × provider × model.
- Logs: structured JSON, with fields for
request_id, client_id, provider, upstream_latency, error_class, fallback_chain. Retain 7 days of raw logs + 90 days of aggregates.
- Traces: OpenTelemetry spans cover "client → gateway → upstream → parse → return," so a single failure pinpoints its exact segment in seconds.
Key Dashboard Panels
- Health quadrants: latency (upstream + gateway), success rate, cost (daily avg + monthly cumulative), remaining quota. Each has an SLO and alert threshold.
- Provider heatmap: success-rate distribution per provider across time slots, used to tune routing weights.
- Prompt length histogram: long prompts risk blowing the context window — catch "user input inflation" early.
Code Example
from opentelemetry import trace, metrics
tracer = trace.get_tracer("gateway")
meter = metrics.get_meter("gateway")
latency_hist = meter.create_histogram("gateway.latency", unit="ms")
token_counter = meter.create_counter("gateway.tokens")
async def call_upstream(req, provider):
with tracer.start_as_current_span(f"upstream.{provider}") as span:
span.set_attribute("model", req["model"])
t0 = time.time()
try:
resp = await http.post(...)
latency_hist.record((time.time()-t0)*1000, {
"provider": provider, "model": req["model"]})
token_counter.add(resp["usage"]["total_tokens"], {
"provider": provider, "dir": "total"})
return resp
except Exception as e:
span.record_exception(e)
span.set_status(trace.Status(trace.StatusCode.ERROR))
raise
Cost vs Observability Balance
Capturing every trace blows up storage; skipping entirely loses debuggability. The balance is tiered sampling: 1% of normal requests, 100% of error requests, 100% of slow requests (above P99 threshold). 99% of traffic never hits disk, but every "worth-seeing" request does. Additionally, tier retention: 7 days raw logs, 90 days aggregates, 1 year cold archive — query performance and storage cost both optimized.
Best Practices
- Sampling: full trace capture explodes storage; sample 1% randomly plus 100% of error requests.
- PII redaction: prompts often contain user PII; scrub email/phone from logs before they hit disk.
- SLO burn rate: use multi-window multi-burn-rate alerting (5m × 1h + 1h × 6h) to catch anomalies earlier than static thresholds.
- Unified ID threading: every request carries matching trace_id / span_id / log_id for cross-table queries.
Observability is not "investigate after the incident" — it is "see the system even when nothing is wrong."