⚠️ 待更新·2026-08-29核验 · 更新时间待核验 · 本文信息可能已过期，请以官方文档为准
更新时间：2026-08-29 · 核验状态：待更新 · 官方溯源待补
背景
免费额度按 RPM/RPD 计,但 30% 的真实流量其实是"同一问题被多人问"。比如"用一句话解释 RAG"被一百个用户各问一次,直接打免费 API 一百次,RPD 当天就被烧光。缓存层把这些重复流量拦下,把免费额度留给真正的新问题。
两层缓存设计
- 精确缓存(L1):
hash(messages + model + temperature) → response,Redis 存 1 小时。命中率通常 20-40%,直接拦截字面重复。
- 语义缓存(L2):把 prompt 转成 embedding,存入向量库。新请求先 embed,检索 top-k 相似 prompt,若余弦相似度 >0.92 且对应响应仍新鲜,直接返回。可拦截"换个说法问同一件事"。
- 预热缓存(L3,可选):对高频问题(产品 FAQ、文档摘要)离线预生成响应,放在 CDN 上,请求连网关都不用打。
命中策略与失效
- 可缓存判定:
temperature=0 且不含 tool_calls 的请求可缓存;创意写作默认不可缓存。
- TTL 分级:事实性问答 24h、代码生成 6h、时效性内容 1h。
- 主动失效:模型版本升级时清空对应 prefix;用户可在响应 header 看到
x-cache: HIT/MISS。
- 缓存 poisoning 防御:对带密钥的 prompt(如
key=xxx)一律不缓存。
代码示例
import hashlib, redis, numpy as np
r = redis.Redis()
from openai import OpenAI
emb_client = OpenAI()
def cache_key(messages, model):
return "v1:" + hashlib.sha256(
(str(messages) + model).encode()).hexdigest()
def get_or_call(messages, model, call_fn):
k = cache_key(messages, model)
if cached := r.get(k):
return cached
# semantic check: embed and search neighbors
emb = emb_client.embeddings.create(
model="text-embedding-3-small",
input=str(messages)).data[0].embedding
if near := vector_db.search(emb, threshold=0.92):
r.setex(k, 3600, near)
return near
resp = call_fn()
r.setex(k, 3600, resp)
vector_db.upsert(emb, resp)
return resp
Embedding 模型选型
语义缓存的命中率取决于 embedding 模型。用 OpenAI text-embedding-3-small(1536 维)命中率好但有成本;用本地 bge-small-zh 零成本但准确率略低。关键陷阱:一旦换了 embedding 模型,向量库全部失效,要重新索引。所以网关要支持"双索引并存"——新模型建立新向量库,旧库继续服务直到命中率为零,然后下线。
最佳实践
- 测量命中率:命中率 <10% 说明缓存策略错了,通常因为 key 设计太粗或 TTL 太短。
- 缓存大小有上限:LRU + 1GB 上限,防止把 Redis 撑爆。
- 可关停:为调试方便,提供
x-bypass-cache: true header 一键绕过。
- 跨用户隔离:不同 client_id 的缓存不要共享,避免敏感数据越过边界。
缓存让免费额度变成"无限"——只要你的问题分布有长尾。
⚠️ Pending Update · 2026-08-29 Verification · Content may be outdated, please refer to official docs
Updated: 2026-08-29 · Status: Pending Verification
Background
Free quota is counted in RPM/RPD, but 30% of real traffic is "the same question asked by many users." For example, "Explain RAG in one sentence" might be asked a hundred times across users; hitting the free API a hundred times burns the daily RPD instantly. A cache layer intercepts this repetition, reserving free quota for genuinely new questions.
Two-Layer Cache Design
- Exact cache (L1):
hash(messages + model + temperature) → response, stored in Redis for 1 hour. Typical hit rate 20-40%, intercepting literal duplicates.
- Semantic cache (L2): embed the prompt, store it in a vector store. New requests embed first, retrieve top-k similar prompts, and if cosine similarity > 0.92 and the cached response is still fresh, return it directly. This catches "same question, different wording."
- Warm cache (L3, optional): pre-generate responses offline for high-frequency queries (product FAQ, doc summaries) and serve from a CDN — these never hit the gateway at all.
Hit Policy and Invalidation
- Cacheability check: requests with
temperature=0 and no tool_calls are cacheable; creative writing is not by default.
- Tiered TTL: 24h for factual Q&A, 6h for code generation, 1h for time-sensitive content.
- Active invalidation: when a model version upgrades, clear the corresponding prefix; users see
x-cache: HIT/MISS in the response header.
- Cache poisoning defense: prompts containing secrets (e.g.
key=xxx) are never cached.
Code Example
import hashlib, redis
r = redis.Redis()
from openai import OpenAI
emb_client = OpenAI()
def cache_key(messages, model):
return "v1:" + hashlib.sha256(
(str(messages) + model).encode()).hexdigest()
def get_or_call(messages, model, call_fn):
k = cache_key(messages, model)
if cached := r.get(k):
return cached
# semantic check: embed and search neighbors
emb = emb_client.embeddings.create(
model="text-embedding-3-small",
input=str(messages)).data[0].embedding
if near := vector_db.search(emb, threshold=0.92):
r.setex(k, 3600, near)
return near
resp = call_fn()
r.setex(k, 3600, resp)
vector_db.upsert(emb, resp)
return resp
Embedding Model Choice
Semantic cache hit rate depends on the embedding model. OpenAI text-embedding-3-small (1536 dims) hits well but has cost; local bge-small-zh is free but slightly less accurate. Key trap: once you change the embedding model, the entire vector store becomes invalid and must be re-indexed. So the gateway must support "dual indexing" — the new model builds a new store while the old one keeps serving until its hit rate reaches zero, then is taken offline.
Best Practices
- Measure hit rate: a rate below 10% usually means the cache key is too coarse or the TTL too short.
- Bound cache size: LRU with a 1GB ceiling to avoid blowing up Redis.
- Provide a bypass: for debugging, expose an
x-bypass-cache: true header that skips the cache.
- Cross-user isolation: never share cache entries across
client_id boundaries to avoid sensitive data leaks.
Caching turns free quota into "unlimited" — as long as your question distribution has a long tail.