usage

NVIDIA NIM 本地与云端点部署

📅 2026-08-19 00:00:00 +0000 UTC 👁 5912 浏览 📂 usage

⚠️ 待更新·2026-08-29核验 · 更新时间待核验 · 本文信息可能已过期，请以官方文档为准更新时间：2026-08-29 · 核验状态：待更新 · 官方溯源待补

简介

NVIDIA NIM(NVIDIA Inference Microservices)是 NVIDIA 推出的容器化推理服务,把 TensorRT-LLM、Triton 等优化打包成 OpenAI 兼容 API。它有两种使用方式:本地用 Docker 跑容器,或直接调用 build.nvidia.com 上的云端端点。本篇分别演示,并给出选型建议。

架构图

flowchart LR A[NVIDIA NIM] --> B[Cloud: build.nvidia.com] A --> C[Local: Docker NIM container] B --> B1[Free 1000 credits] C --> C1[Self-hosted GPU box] B1 --> D[OpenAI-compatible endpoint] C1 --> D

方式一:本地容器部署

前置条件:NVIDIA GPU(显存 ≥ 16GB)+ 驱动 ≥ 535、nvidia-container-toolkit、NGC 账号。

# 1. 登录 NGC 拉取镜像
docker login nvcr.io -u '$oauthtoken' -p "$(cat ngc-key.txt)"

# 2. 拉取 Llama-3.1-8B NIM 镜像
docker pull nvcr.io/nim/meta/llama-3.1-8b-instruct:1.0.0

# 3. 启动容器(单卡)
docker run -d --name nim-llama \
  --runtime=nvidia --gpus all \
  -e NGC_API_KEY=$(cat ngc-key.txt) \
  -v $HOME/.cache/nim:/opt/nim/.cache \
  -p 8000:8000 \
  --shm-size=16g \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:1.0.0

首次启动会下载模型权重(约 16GB),完成后访问 http://localhost:8000/docs 查看 Swagger 文档,/v1/models 列出可用模型,/v1/chat/completions 是 OpenAI 兼容端点。

方式二:调用云端端点

不想自己装 GPU?直接用 build.nvidia.com 托管的端点。在 https://build.nvidia.com 注册并生成 NGC API Key,然后即可调用:

curl https://integrate.api.nvidia.com/v1/chat/completions \
  -H "Authorization: Bearer $NVIDIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [{"role":"user","content":"hi"}],
    "max_tokens": 128
  }'

新账号有 1000 次免费调用额度,适合快速原型验证。云端还托管很多特色模型,如 nvidia/llama-3.1-nemotron-70b-instruct(NVIDIA 自研对齐版本)、deepseek-ai/deepseek-r1(推理模型)。

选型建议

维度	本地 NIM	云端 NIM
启动成本	高(需 A10/A100)	0
延迟	低(本地 GPU)	中(走公网)
数据隐私	完全本地	数据出网
适用场景	内网/合规场景	原型/小流量

性能调优

开启 TensorRT-LLM:NIM 镜像默认启用,首次启动会编译引擎(约 5 分钟),之后推理提速 2-5 倍。
调整 batch size:在容器环境变量里设 NIM_MAX_BATCH_SIZE=128 提升吞吐。
多卡推理:--gpus all + NIM_TENSOR_PARALLEL_SIZE=2 把模型切到两张卡上。

常见问题

CUDA out of memory:8B 模型至少需要 16GB 显存,加 --shm-size=16g 重试,或换 4B 模型。
拉镜像 401:确认 NGC API Key 有效,且账号已加入 NIM early access。
想跑别的模型:在 https://build.nvidia.com 浏览所有可用 NIM 镜像,替换 nvcr.io/nim/<vendor>/<model> 即可。
冷启动慢:容器首次启动需下载并编译引擎,生产环境用 --restart unless-stopped 保活。

部署完成后,参考《NVIDIA NIM 推理调用示例》用 OpenAI SDK 调用。

部署对比

部署方式	优势	限制
云端 build.nvidia.com	零运维，1000 credits	共享 QPS，延迟有波动
本地 Docker NIM	独占 GPU，低延迟	需 GPU 卡（A10/L4 起步）

最佳实践

首次跑用云端：1000 credits 跑完调通流程，再决定是否上本地。
本地选 L4 而非 A10：L4 性价比高，单卡跑 7B-70B 量化版都够。
NIM 容器要 GPU 驱动：装好 nvidia-docker runtime，否则容器看不到 GPU。

⚠️ Pending Update · 2026-08-29 Verification · Content may be outdated, please refer to official docs Updated: 2026-08-29 · Status: Pending Verification

Introduction

NVIDIA NIM (NVIDIA Inference Microservices) is a containerized inference service that bundles optimizations like TensorRT-LLM and Triton behind an OpenAI-compatible API. You can either run NIM locally with Docker or call hosted endpoints on build.nvidia.com. This article demonstrates both and gives guidance on choosing between them.

架构图

flowchart LR A[NVIDIA NIM] --> B[Cloud: build.nvidia.com] A --> C[Local: Docker NIM container] B --> B1[Free 1000 credits] C --> C1[Self-hosted GPU box] B1 --> D[OpenAI-compatible endpoint] C1 --> D

Option 1: Local Container Deployment

Prerequisites: an NVIDIA GPU (>= 16 GB VRAM) with driver >= 535, nvidia-container-toolkit, and an NGC account.

# 1. Log in to NGC and pull the image
docker login nvcr.io -u '$oauthtoken' -p "$(cat ngc-key.txt)"

# 2. Pull the Llama-3.1-8B NIM image
docker pull nvcr.io/nim/meta/llama-3.1-8b-instruct:1.0.0

# 3. Start the container (single GPU)
docker run -d --name nim-llama \
  --runtime=nvidia --gpus all \
  -e NGC_API_KEY=$(cat ngc-key.txt) \
  -v $HOME/.cache/nim:/opt/nim/.cache \
  -p 8000:8000 \
  --shm-size=16g \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:1.0.0

The first launch downloads the model weights (~16 GB). When ready, visit http://localhost:8000/docs for the Swagger UI. /v1/models lists available models, and /v1/chat/completions is the OpenAI-compatible endpoint.

Option 2: Call Cloud Endpoints

Skip the GPU and use the hosted endpoints on build.nvidia.com. Register at https://build.nvidia.com, generate an NGC API key, then call:

curl https://integrate.api.nvidia.com/v1/chat/completions \
  -H "Authorization: Bearer $NVIDIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [{"role":"user","content":"hi"}],
    "max_tokens": 128
  }'

New accounts receive 1000 free calls — perfect for quick prototyping. The cloud also hosts specialized models like nvidia/llama-3.1-nemotron-70b-instruct (NVIDIA's aligned variant) and deepseek-ai/deepseek-r1 (a reasoning model).

Choosing Between Modes

Dimension	Local NIM	Cloud NIM
Setup cost	High (needs A10/A100)	None
Latency	Low (local GPU)	Medium (over the internet)
Privacy	Fully on-prem	Data leaves the network
Best for	Internal/compliance workloads	Prototyping / low traffic

Performance Tuning

Enable TensorRT-LLM: enabled by default in NIM images. The first launch compiles the engine (~5 min), after which inference is 2-5x faster.
Tune batch size: set NIM_MAX_BATCH_SIZE=128 in container env to boost throughput.
Multi-GPU inference: use --gpus all plus NIM_TENSOR_PARALLEL_SIZE=2 to shard the model across two GPUs.

Troubleshooting

CUDA out of memory: The 8B model needs at least 16 GB of VRAM. Retry with --shm-size=16g or switch to a 4B model.
401 on image pull: Verify your NGC API key is valid and your account is enrolled in the NIM early access program.
Other models: Browse all NIM images at https://build.nvidia.com and swap nvcr.io/nim/<vendor>/<model> accordingly.
Slow cold start: The first start downloads weights and compiles the engine. Use --restart unless-stopped in production to keep the container warm.

Once deployed, see "NVIDIA NIM Inference Example" to call it via the OpenAI SDK.

Deployment Comparison

Path	Strength	Limitation
Cloud build.nvidia.com	Zero ops, 1000 credits	Shared QPS, variable latency
Local Docker NIM	Dedicated GPU, low latency	Needs GPU card (A10/L4 minimum)

Best Practices

Start with cloud: burn through the 1000 credits to validate the flow before deciding on local.
Pick L4 over A10 locally: L4 has better price/perf; a single card runs 7B-70B quantized comfortably.
GPU driver prerequisite: install nvidia-docker runtime, otherwise the container cannot see the GPU.