⚠️ 待更新·2026-08-29核验 · 更新时间待核验 · 本文信息可能已过期，请以官方文档为准
更新时间：2026-08-29 · 核验状态：待更新 · 官方溯源待补
背景
医疗、金融、政务场景下,数据不允许出内网;离线/边缘设备(Satcom、工厂车间)也无法访问公网。把开源模型用 Ollama / vLLM 部署在本地,作为统一网关的"内部 Provider",与云端免费模型并列,既能满足合规,又能在断网时降级。
核心架构
- 混合 Provider 池:本地
ollama/llama3 与云端 openrouter/deepseek 同列路由表。网关按数据敏感度标签路由:打标 sensitive 的请求强制走本地。
- 本地 Provider 接入:Ollama 暴露
http://localhost:11434/v1,vLLM 暴露 http://localhost:8000/v1,两者都原生 OpenAI 兼容,接入零改造。
- 离线降级链:
cloud-primary → local-fallback → cache-only。公网可用时优先云端免费,断网时切本地,本地也跑不动时返回缓存。
- 资源治理:本地 GPU 是稀缺资源,设并发上限(如 2 路 concurrent),排队超 30s 自动溢出到云端。
代码示例
PROVIDERS = {
"local": {"base": "http://localhost:11434/v1", "key": "ollama",
"sensitive_only": True, "max_concurrent": 2},
"cloud": {"base": "https://openrouter.ai/api/v1", "key": "sk-or-..."},
}
sem = asyncio.Semaphore(PROVIDERS["local"]["max_concurrent"])
async def route(req, is_sensitive):
if is_sensitive:
async with sem:
return await call(PROVIDERS["local"], req)
try:
return await call(PROVIDERS["cloud"], req)
except (NetworkError, TimeoutError):
# offline fallback to local
async with sem:
return await call(PROVIDERS["local"], req)
模型版本锁定
本地部署的模型版本必须显式锁定。Ollama 默认拉 latest,但 llama3:latest 上周和这周可能是不同的微调版本,会让线上行为漂移。生产环境必须用 llama3:8b-instruct-q5_k_M 这样的全 tag 锁定。同时维护一个"模型清单"YAML,记录每家本地模型的精确版本、量化方法、上下文长度,变更走 PR Review,杜绝"今天调的好好的,明天模型自己变了"。
最佳实践
- 本地模型选型:推理任务用 7B(本地单卡可跑),复杂任务用 70B(本地多卡或量化),按
model_size 字段路由。
- 数据脱敏:即使走本地,日志里也别记明文 prompt,防止日志泄露。
- 资源监控:本地 GPU 利用率、显存、队列深度,纳入网关指标看板,别让本地 Provider 变成隐形瓶颈。
- 灰度:新部署的本地 Provider 先承接 1% 流量,观察成功率与延迟,再逐步切量。
- 冷启动预热:本地大模型首次加载需要 30s+,网关启动后主动发个 ping 预热,避免首个真实请求被冷启动拖慢。
私有化不是"只能本地",而是"敏感的本地、普通的云端、网关层做总指挥"。
⚠️ Pending Update · 2026-08-29 Verification · Content may be outdated, please refer to official docs
Updated: 2026-08-29 · Status: Pending Verification
Background
In healthcare, finance, and government scenarios, data cannot leave the intranet; offline or edge devices (satcom, factory floors) have no public-internet access at all. Deploying open-source models locally via Ollama or vLLM, then registering them as "internal providers" alongside cloud free models in the unified gateway, satisfies both compliance and offline degradation requirements.
Core Architecture
- Hybrid provider pool: local
ollama/llama3 and cloud openrouter/deepseek both sit in the routing table. The gateway routes by data-sensitivity tag — sensitive requests are forced to the local pool.
- Local provider onboarding: Ollama exposes
http://localhost:11434/v1 and vLLM exposes http://localhost:8000/v1; both are natively OpenAI-compatible, so onboarding needs zero code changes.
- Offline degradation chain:
cloud-primary → local-fallback → cache-only. When the internet is up, prefer cloud free; when offline, switch to local; when local cannot run either, return cached responses.
- Resource governance: the local GPU is scarce — cap concurrency (e.g. 2 concurrent) and spill to cloud after a 30s queue.
Code Example
PROVIDERS = {
"local": {"base": "http://localhost:11434/v1", "key": "ollama",
"sensitive_only": True, "max_concurrent": 2},
"cloud": {"base": "https://openrouter.ai/api/v1", "key": "sk-or-..."},
}
sem = asyncio.Semaphore(PROVIDERS["local"]["max_concurrent"])
async def route(req, is_sensitive):
if is_sensitive:
async with sem:
return await call(PROVIDERS["local"], req)
try:
return await call(PROVIDERS["cloud"], req)
except (NetworkError, TimeoutError):
# offline fallback to local
async with sem:
return await call(PROVIDERS["local"], req)
Model Version Pinning
Locally deployed models must be explicitly version-pinned. Ollama pulls latest by default, but llama3:latest last week and this week may be different fine-tunes, causing production drift. Production must use full tags like llama3:8b-instruct-q5_k_M. Maintain a "model manifest" YAML recording each local model's exact version, quantization method, and context length; changes go through PR review — never let "it worked yesterday, the model changed itself today" happen.
Best Practices
- Local model selection: use a 7B model for inference tasks (runs on a single local GPU) and a 70B model (multiple GPUs or quantized) for complex tasks; route by
model_size.
- Data redaction: even when traffic stays local, do not log plaintext prompts — log leakage remains a risk.
- Resource monitoring: track local GPU utilization, VRAM, and queue depth in the gateway metrics dashboard so the local provider does not become an invisible bottleneck.
- Canary: a newly deployed local provider first absorbs 1% of traffic, with success rate and latency under observation, before ramping up.
- Cold-start warm-up: a local large model needs 30s+ on first load; the gateway pings it on startup to pre-warm so the first real request is not stuck behind a cold start.
Private deployment is not "local only" — it is "sensitive local, ordinary cloud, gateway as conductor."