⚠️ 待更新·2026-08-29核验 · 更新时间待核验 · 本文信息可能已过期，请以官方文档为准
更新时间：2026-08-29 · 核验状态：待更新 · 官方溯源待补
简介
NIM 端点完全兼容 OpenAI Chat Completions 协议,这意味着你已经熟悉的 openai Python SDK 可以直接复用,只需改 base_url 和 api_key。本篇给出四种最常用调用形态:单轮、流式、多轮、JSON 结构化输出。
架构图
flowchart TD
A[OpenAI SDK] --> B[NIM endpoint]
B --> C[Chat completion]
B --> D[Streaming]
B --> E[Multi-turn]
B --> F[JSON mode]
C --> G[Response]
D --> G
E --> G
F --> G
准备
pip install openai
设置环境变量(本地或云端任选其一):
# 本地 NIM 容器
export NIM_BASE_URL="http://localhost:8000/v1"
export NIM_API_KEY="local-no-key-needed"
# 云端 build.nvidia.com
export NIM_BASE_URL="https://integrate.api.nvidia.com/v1"
export NIM_API_KEY="nvapi-..."
示例 1:单轮对话
import os
from openai import OpenAI
client = OpenAI(
base_url=os.environ["NIM_BASE_URL"],
api_key=os.environ["NIM_API_KEY"],
)
resp = client.chat.completions.create(
model="meta/llama-3.1-8b-instruct",
messages=[
{"role": "system", "content": "你是简洁的中文助手"},
{"role": "user", "content": "用一句话介绍量子纠缠"},
],
temperature=0.5,
max_tokens=128,
)
print(resp.choices[0].message.content)
print("tokens:", resp.usage.total_tokens)
示例 2:流式输出
长回答场景下流式可显著改善首字延迟:
stream = client.chat.completions.create(
model="meta/llama-3.1-8b-instruct",
messages=[{"role": "user", "content": "写一首关于秋天的五言绝句"}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print()
示例 3:多轮对话
history = [
{"role": "system", "content": "你是耐心的编程老师"}
]
def chat(user_text):
history.append({"role": "user", "content": user_text})
r = client.chat.completions.create(
model="meta/llama-3.1-8b-instruct",
messages=history,
)
msg = r.choices[0].message
history.append(msg)
return msg.content
print(chat("Python 里 list 和 tuple 有什么区别?"))
print(chat("那什么时候该用 tuple?"))
示例 4:结构化 JSON 输出
需要让模型返回可解析的 JSON 时,显式声明 response_format:
import json
resp = client.chat.completions.create(
model="meta/llama-3.1-8b-instruct",
messages=[
{"role": "system", "content": "你是数据抽取助手,只输出 JSON。"},
{"role": "user", "content": "从这句话抽取人名和职位:张三是阿里巴巴的高级工程师。"},
],
response_format={"type": "json_object"},
max_tokens=128,
)
data = json.loads(resp.choices[0].message.content)
print(data)
# {'name': '张三', 'title': '高级工程师', 'company': '阿里巴巴'}
性能调优
- 开启 streaming:
stream=True 让首字延迟降到 100ms 内。
- 复用 client:把
OpenAI() 实例做成单例,避免每次请求重建连接池。
- 设置超时:
client = OpenAI(..., timeout=30, max_retries=2) 防止长尾请求卡死。
- 批量请求:NIM 支持
/v1/batch 端点(部分模型),适合离线处理。
进阶用法:工具调用与批处理
NIM 也支持 OpenAI 风格的 function calling,适合做 Agent 编排:
tools = [{
"type": "function",
"function": {
"name": "get_stock",
"description": "查询股票价格",
"parameters": {
"type": "object",
"properties": {"symbol": {"type": "string"}},
"required": ["symbol"],
},
},
}]
resp = client.chat.completions.create(
model="meta/llama-3.1-8b-instruct",
messages=[{"role":"user","content":"查 AAPL"}],
tools=tools,
)
print(resp.choices[0].message.tool_calls[0].function)
对于离线批量任务,部分 NIM 镜像还提供 /v1/batch 端点,可一次性提交上千条 prompt,按完成度计费,成本比逐条调用低 50%。
常见问题
model not found:本地 NIM 容器的模型名是镜像内置的,通过 GET /v1/models 查询实际名称。- 云端响应慢:首次冷启动可能 1-2 秒,后续同会话稳定在百毫秒级。
- JSON 解析失败:在 system prompt 里强约束格式,如 "只输出 JSON,字段为 name/title/company"。
掌握这四种调用,绝大多数业务场景已覆盖。
最佳实践
- 流式优先：NIM 流式 token 生成速度比非流式整体快 30%，用户体感更好。
- JSON mode 用于结构化输出：调用
response_format={"type":"json_object"} 强制 JSON，避免解析失败。
- 温度参数控制随机性：默认 0.7 适合对话，写代码改 0.3，做摘要改 0.5。
- max_tokens 留 1024 余量：模型常用 max_tokens 截断长回答，留余量避免被截。
⚠️ Pending Update · 2026-08-29 Verification · Content may be outdated, please refer to official docs
Updated: 2026-08-29 · Status: Pending Verification
Introduction
NIM endpoints are fully OpenAI Chat Completions-compatible, which means you can reuse the familiar openai Python SDK — just swap the base_url and api_key. This article covers four common patterns: single-turn, streaming, multi-turn, and structured JSON output.
架构图
flowchart TD
A[OpenAI SDK] --> B[NIM endpoint]
B --> C[Chat completion]
B --> D[Streaming]
B --> E[Multi-turn]
B --> F[JSON mode]
C --> G[Response]
D --> G
E --> G
F --> G
Setup
pip install openai
Set environment variables (local or cloud — pick one):
# Local NIM container
export NIM_BASE_URL="http://localhost:8000/v1"
export NIM_API_KEY="local-no-key-needed"
# Cloud build.nvidia.com
export NIM_BASE_URL="https://integrate.api.nvidia.com/v1"
export NIM_API_KEY="nvapi-..."
Example 1: Single-turn Chat
import os
from openai import OpenAI
client = OpenAI(
base_url=os.environ["NIM_BASE_URL"],
api_key=os.environ["NIM_API_KEY"],
)
resp = client.chat.completions.create(
model="meta/llama-3.1-8b-instruct",
messages=[
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "Explain quantum entanglement in one sentence."},
],
temperature=0.5,
max_tokens=128,
)
print(resp.choices[0].message.content)
print("tokens:", resp.usage.total_tokens)
Example 2: Streaming
Streaming dramatically reduces time-to-first-token for long answers:
stream = client.chat.completions.create(
model="meta/llama-3.1-8b-instruct",
messages=[{"role": "user", "content": "Write a haiku about autumn."}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print()
Example 3: Multi-turn Conversation
history = [
{"role": "system", "content": "You are a patient programming tutor."}
]
def chat(user_text):
history.append({"role": "user", "content": user_text})
r = client.chat.completions.create(
model="meta/llama-3.1-8b-instruct",
messages=history,
)
msg = r.choices[0].message
history.append(msg)
return msg.content
print(chat("What is the difference between list and tuple in Python?"))
print(chat("When should I prefer a tuple?"))
Example 4: Structured JSON Output
When you need machine-parseable JSON, declare response_format explicitly:
import json
resp = client.chat.completions.create(
model="meta/llama-3.1-8b-instruct",
messages=[
{"role": "system", "content": "You are a data extraction assistant. Output JSON only."},
{"role": "user", "content": "Extract name and title: 'Alice is a Senior Engineer at Acme Corp.'"},
],
response_format={"type": "json_object"},
max_tokens=128,
)
data = json.loads(resp.choices[0].message.content)
print(data)
# {'name': 'Alice', 'title': 'Senior Engineer', 'company': 'Acme Corp.'}
- Enable streaming:
stream=True brings time-to-first-token under 100ms.
- Reuse the client: make
OpenAI() a singleton to avoid rebuilding the connection pool per request.
- Set timeouts:
client = OpenAI(..., timeout=30, max_retries=2) to prevent long-tail hangs.
- Batch requests: NIM supports
/v1/batch (some models) for offline workloads.
NIM also supports OpenAI-style function calling, ideal for agent orchestration:
tools = [{
"type": "function",
"function": {
"name": "get_stock",
"description": "Get stock price",
"parameters": {
"type": "object",
"properties": {"symbol": {"type": "string"}},
"required": ["symbol"],
},
},
}]
resp = client.chat.completions.create(
model="meta/llama-3.1-8b-instruct",
messages=[{"role":"user","content":"Check AAPL"}],
tools=tools,
)
print(resp.choices[0].message.tool_calls[0].function)
For offline batch jobs, some NIM images also expose /v1/batch so you can submit thousands of prompts at once, billed by completion — about 50% cheaper than per-call.
Troubleshooting
model not found: Local NIM model names are baked into the image. Query GET /v1/models to see the actual ID.- Slow cloud responses: The first call may take 1-2 seconds for cold start; subsequent calls settle to hundreds of milliseconds.
- JSON parse failures: Strongly constrain the format in the system prompt, e.g. "Output JSON only with fields name/title/company".
Master these four patterns and you have most business scenarios covered.
Best Practices
- Prefer streaming: NIM streaming is ~30% faster than non-streaming overall, with better UX.
- Use JSON mode for structured output: pass
response_format={"type":"json_object"} to force JSON and avoid parse failures.
- Tune temperature for the task: 0.7 default suits chat; 0.3 for code; 0.5 for summarization.
- Leave 1024-token headroom in max_tokens: models often truncate long answers; leave headroom to avoid being cut off.