usage

Groq 极速推理接入教程

📅 2026-08-19 00:00:00 +0000 UTC 👁 4881 浏览 📂 usage

⚠️ 待更新·2026-08-29核验 · 更新时间待核验 · 本文信息可能已过期，请以官方文档为准更新时间：2026-08-29 · 核验状态：待更新 · 官方溯源待补

简介

Groq 是一家专注推理加速的硬件公司,自研的 LPU(Language Processing Unit)在 Llama 系列模型上能跑到 500+ tokens/s,比常见 GPU 快一两个数量级。更重要的是,它提供慷慨的免费层,且完全兼容 OpenAI 协议。本篇三步接入并演示流式与工具调用。

架构图

flowchart LR A[Sign up at console.groq.com] --> B[Create API Key] B --> C[Install openai SDK] C --> D[Call llama-3.3-70b-versatile] D --> E[500+ tokens/s]

第一步:申请 Key

访问 https://console.groq.com,用 Google 或 GitHub 登录。
进入 API Keys → Create API Key,复制 gsk_... 开头的字符串。
设置环境变量:

export GROQ_API_KEY="gsk_..."

第二步:安装 SDK

Groq 官方维护 groq Python SDK,也可直接用 openai SDK。推荐前者,类型提示更完整:

pip install groq

第三步:发起调用

import os
from groq import Groq

client = Groq(api_key=os.environ["GROQ_API_KEY"])

resp = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[
        {"role": "system", "content": "你是简洁的中文助手"},
        {"role": "user", "content": "用一句话解释什么是反向索引"},
    ],
    temperature=0.3,
    max_tokens=256,
)
print(resp.choices[0].message.content)

流式输出

体验 LPU 真实速度的最佳方式就是流式,首字延迟常低于 200ms:

stream = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "写一首关于秋天的五言绝句"}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()

工具调用(Tool Use)

Groq 支持 OpenAI 风格的 function calling,适合做 Agent:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "查询某城市天气",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
resp = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "北京今天天气怎么样?"}],
    tools=tools,
)
call = resp.choices[0].message.tool_calls[0]
print(call.function.name, call.function.arguments)
# get_weather {"city": "北京"}

免费层限制(截至 2025)

每分钟请求数:30 RPM(70B 模型)、1440 RPM(8B 模型)
每日 token 上限:约 1M tokens/day,按模型不同浮动
上下文窗口:Llama 3.3 70B 默认 128K

速度实测与对比

在 Llama-3.3-70B 上实测,Groq LPU 单请求生成速度可达 250-500 tokens/s,而同模型在 GPU 推理服务上通常只有 30-80 tokens/s。对延迟敏感的实时聊天、代码补全、Agent 多轮调用场景,这种速度差距能直接转化为用户体验提升。需要注意的是,LPU 对长输入(>8K)的 prefill 阶段优势会缩小,此时 GPU 的高带宽反而占优。

常见问题

429 rate_limit_exceeded:免费层触顶,等 60 秒或换更小的 llama-3.1-8b-instant 模型。
想用 JSON mode:加 response_format={"type":"json_object"},模型必须被提示输出 JSON。
可用模型清单:访问 https://api.groq.com/openai/v1/models 查看当前支持的所有模型。
响应被截断:max_tokens 默认较小,设为 1024+ 可解决。

Groq 适合做实时对话、代码补全这类对延迟敏感的场景。

最佳实践

batch 请求：Groq 支持 batch（一条请求多轮对话），单次吞吐量 3-5 倍。
避开 30 RPM 限速：用 token bucket 算法本地限速到 25 RPM，留 buffer。
流式优先：500+ tps 的流式输出体验远好于等整段。
Mixtral 8x7B 在 Groq 上最猛：MoE 模型 LPU 加速效果比 dense 模型更明显。

⚠️ Pending Update · 2026-08-29 Verification · Content may be outdated, please refer to official docs Updated: 2026-08-29 · Status: Pending Verification

Introduction

Groq is a hardware company focused on inference acceleration. Its LPU (Language Processing Unit) runs Llama-family models at 500+ tokens/s — one to two orders of magnitude faster than typical GPUs. Better yet, it offers a generous free tier and is fully OpenAI-compatible. This article gets you connected in three steps and demos streaming and tool use.

架构图

flowchart LR A[Sign up at console.groq.com] --> B[Create API Key] B --> C[Install openai SDK] C --> D[Call llama-3.3-70b-versatile] D --> E[500+ tokens/s]

Step 1: Get an API Key

Visit https://console.groq.com and sign in with Google or GitHub.
Go to API Keys → Create API Key and copy the string starting with gsk_....
Export it:

export GROQ_API_KEY="gsk_..."

Step 2: Install the SDK

Groq maintains an official groq Python SDK. You can also use the openai SDK, but the former has better type hints:

pip install groq

Step 3: Make a Call

import os
from groq import Groq

client = Groq(api_key=os.environ["GROQ_API_KEY"])

resp = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Explain an inverted index in one sentence."},
    ],
    temperature=0.3,
    max_tokens=256,
)
print(resp.choices[0].message.content)

Streaming

The best way to feel LPU speed is streaming — first-token latency is often under 200ms:

stream = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Write a haiku about autumn."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()

Tool Use

Groq supports OpenAI-style function calling, ideal for agent workflows:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
resp = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "What's the weather in Tokyo today?"}],
    tools=tools,
)
call = resp.choices[0].message.tool_calls[0]
print(call.function.name, call.function.arguments)
# get_weather {"city": "Tokyo"}

Free-tier Limits (as of 2025)

Requests per minute: 30 RPM (70B), 1440 RPM (8B)
Daily token cap: ~1M tokens/day, varying by model
Context window: 128K for Llama 3.3 70B

Real-world Speed Benchmarks

On Llama-3.3-70B, Groq's LPU sustains 250-500 tokens/s per request, whereas the same model on typical GPU inference services only reaches 30-80 tokens/s. For latency-sensitive use cases — real-time chat, code completion, multi-turn agents — this gap translates directly into better UX. Note that for long inputs (>8K tokens) the LPU's advantage narrows in the prefill phase, where GPU memory bandwidth wins.

Troubleshooting

429 rate_limit_exceeded: Free-tier cap hit. Wait 60 seconds or switch to the smaller llama-3.1-8b-instant model.
JSON mode: Add response_format={"type":"json_object"} and prompt the model to output JSON.
List available models: Visit https://api.groq.com/openai/v1/models.
Truncated responses: max_tokens defaults low — set it to 1024+.

Groq is ideal for latency-sensitive use cases like real-time chat and code completion.

Best Practices

Use batch requests: Groq supports batch (one request, multiple turns) — 3-5x throughput.
Stay below 30 RPM: implement a local token bucket at 25 RPM to leave buffer.
Prefer streaming: 500+ tps streaming UX beats waiting for the full response.
Mixtral 8x7B is fastest on Groq: MoE models benefit more from LPU acceleration than dense models.