Running GLM-5.2 in Production with vLLM: Quants, GPU Pods, and Verified Serving Recipes

GLM-5.2 is Z.ai’s 743B-parameter Mixture-of-Experts model for coding, reasoning, and long-running agent workflows. Only about 39B parameters are active for each token, and the model supports a native context window of 1,048,576 tokens.

If you are renting GPUs to run it yourself, two parts of the architecture matter more than the raw parameter count.

The first is IndexShare. Instead of running a separate sparse-attention indexer in every layer, GLM-5.2 reuses one indexer across groups of four layers. Z.ai reports that this cuts per-token FLOPs by about 2.9× at 1M context.

The second is Multi-Token Prediction, or MTP. GLM-5.2 can draft up to five tokens ahead during decoding. Z.ai reports roughly a 20% increase in accepted draft length over the previous GLM generation. Whether that produces a worthwhile speedup on your server still depends on how often those draft tokens are accepted on your actual prompts.

For most vLLM deployments, start with:

zai-org/GLM-5.2-FP8

The FP8 checkpoint is much easier to deploy than the full BF16 model and has the broadest set of published GPU recipes.

Which checkpoint fits your hardware?

The main choices are:

If you simply want to get GLM-5.2 running, FP8 is the least complicated starting point.

The full BF16 checkpoint needs a larger multi-node configuration and extra loader setup, so there is little reason to begin there unless BF16 is specifically required.

NVIDIA’s NVFP4 checkpoint is interesting on Blackwell because it reduces the model footprint substantially. AMD has a separate MXFP4 checkpoint for MI355X.

Set up vLLM before renting a long-lived pod

The current GLM-5.2 recipe uses:

vLLM 0.23.0

as its stable baseline.

Install it with:

uv venv
source .venv/bin/activate

uv pip install "vllm==0.23.0" --torch-backend=auto
uv pip install "transformers>=5.9.0"

There are also dedicated Docker images:

vllm/vllm-openai:glm52

or, for CUDA 12.x:

vllm/vllm-openai:glm52-cu129

A basic launch looks like:

docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:glm52 \
  zai-org/GLM-5.2-FP8 \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name glm-5.2-fp8 \
  --max-model-len 131072

For good FP8 performance, the current recipe also expects DeepGEMM. Install it using the install_deepgemm.sh helper supplied by the vLLM setup.

One version caveat is worth knowing before debugging speculative decoding: if you need tool calling and MTP at the same time, the current vLLM guidance points to the latest main branch because not every relevant fix is present in 0.23.0.

Which GPUs should you rent?

These are the useful upstream-backed configurations:

H100, A100, 4× H200, and smaller B200 configurations are deliberately not included here. They may be possible, but the current GLM-5.2 recipe does not provide an equivalent end-to-end deployment path for them.

That distinction matters if you are paying by the hour. A configuration that looks plausible from VRAM alone is not the same thing as one with a published runtime, kernel path, and serving command.

8× H200 is the straightforward NVIDIA setup

For normal FP8 serving on Hopper, the published recipe uses TP8, FP8 KV cache, and GLM-5.2’s five-token MTP head:

vllm serve zai-org/GLM-5.2-FP8 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name glm-5.2-fp8

If you want a conventional NVIDIA pod and do not need the full million-token context window, this is the cleanest place to start.

H20 has a separate DSpark path

On 8× H20, vLLM publishes a recipe using an external DSpark speculator rather than the built-in MTP head:

vllm serve zai-org/GLM-5.2-FP8 \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --max-model-len 200000 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 32768 \
  --speculative-config '{"model":"RedHatAI/GLM-5.2-speculator.dspark","num_speculative_tokens":7,"method":"dspark"}' \
  --served-model-name glm-5.2-fp8

vLLM reports 70+ tokens/sec for a single request on this setup.

Treat that as a result from their exact configuration, not as the number every H20 rental will produce.

If you actually need 1M context, use the B200 recipe

The important full-context NVIDIA setup is:

8× B200
GLM-5.2-FP8
FP8 KV cache

Run:

VLLM_DEEP_GEMM_WARMUP=skip \
vllm serve zai-org/GLM-5.2-FP8 \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --max-num-seqs 32 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name glm-5.2-fp8

The setting to pay attention to here is:

--max-num-seqs 32

Every active request competes for the same KV-cache budget. If long-context requests start running out of memory, lower this value. If your prompts are shorter, you can usually afford more concurrency.

FP8 KV cache is also a major part of this configuration:

--kv-cache-dtype fp8_e4m3

It cuts the cache footprint enough to make the million-token window practical on the published B200 setup.

VLLM_DEEP_GEMM_WARMUP=skip reduces startup time by skipping the initial DeepGEMM JIT warmup. Kernels then compile as requests begin arriving.

Blackwell users can trade FP8 for NVFP4

NVIDIA publishes:

nvidia/GLM-5.2-NVFP4

for B200 and B300.

This is an NVIDIA Model Optimizer conversion, not a random community quant. The MoE expert linear layers move to NVFP4, while components such as the shared expert, attention, embeddings, and other parts of the model remain at higher precision.

Serve it with:

vllm serve nvidia/GLM-5.2-NVFP4 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --trust-remote-code \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --kv-cache-dtype fp8_e4m3 \
  --host 0.0.0.0 \
  --port 8000

You do not need:

--quantization nvfp4

because vLLM reads the quantization configuration from the checkpoint itself.

If you are choosing between FP8 and NVFP4 on Blackwell, NVFP4 is the option to test when model footprint matters more than sticking to the native FP8 release.

Running GLM-5.2 on AMD

For MI300X and MI355X, the FP8 route uses AMD’s AITER kernels:

VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
vllm serve zai-org/GLM-5.2-FP8 \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --gpu-memory-utilization 0.80 \
  --max-model-len 524288 \
  --max-num-seqs 32 \
  --linear-backend aiter \
  --moe-backend aiter

The published starting context here is 524,288 tokens. That is not the model’s maximum; GLM-5.2 still supports 1M. Increase it only after confirming that the server has enough KV-cache headroom.

The 0.80 GPU-memory utilization value is also intentional. It leaves space for ROCm runtime allocations and MTP graph capture instead of trying to occupy virtually every byte of HBM on startup.

MI355X also has an MXFP4 option

AMD publishes:

amd/GLM-5.2-MXFP4

and the current recipe uses 8× MI355X with TP8:

export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_USE_AITER_FP8BMM=0
export VLLM_ROCM_USE_AITER_FP4BMM=0

vllm serve amd/GLM-5.2-MXFP4 \
  --trust-remote-code \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --no-enable-prefix-caching \
  --served-model-name glm-5.2-mxfp4

There is a significant tradeoff here: the published MXFP4 recipe uses only a 32K context window.

If you are choosing GLM-5.2 specifically because you need hundreds of thousands of tokens of context, use the FP8 ROCm recipe instead.

AMD reports 93.93 on GSM8K versus 94.09 for the base model, or roughly 99.8% score recovery, for this MXFP4 conversion.

The MTP layer itself remains at higher precision. The published command simply does not enable speculative decoding, so do not read the absence of --speculative-config as evidence that MTP has been removed from the checkpoint.

Don't allocate 1M context just because the model supports it

GLM-5.2 can handle:

1,048,576 tokens

but there is no reason for every endpoint to reserve enough memory for million-token requests.

The current recipes themselves use very different context limits:

A coding agent working over a giant repository may justify 500K or 1M context. A normal API dominated by 10K prompts probably does not.

Longer context consumes KV cache that could otherwise serve more users, so choose --max-model-len around the traffic you expect rather than the largest number on the model card.

Reasoning and tool calling

Thinking is enabled by default.

GLM-5.2 supports three useful modes:

Think Max
Think High
Non-thinking

Think Max is the default.

For Think High:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5.2-fp8",
    "messages": [
      {
        "role": "user",
        "content": "Design a fault-tolerant distributed queue."
      }
    ],
    "temperature": 1,
    "max_tokens": 4096,
    "chat_template_kwargs": {
      "reasoning_effort": "high"
    }
  }'

For a fast non-thinking response, pass:

{
  "enable_thinking": false
}

through chat_template_kwargs.

If you are building an agent or tool-using application, start vLLM with:

--tool-call-parser glm47 \
--reasoning-parser glm45 \
--enable-auto-tool-choice

Which GLM-5.2 setup should you choose?

If you are renting GPUs specifically for GLM-5.2, the current upstream paths make the decision fairly simple.

8× H200 + FP8 is the straightforward NVIDIA setup.

8× H20 + FP8 is the documented choice if you specifically want the DSpark serving path.

If you really need the full 1M context window, use 8× B200 + FP8.

On Blackwell, 8× B200 or B300 + NVIDIA NVFP4 is the lower-footprint alternative.

For AMD, use 8× MI300X or MI355X + FP8 when long context matters. Use 8× MI355X + MXFP4 when reducing the model footprint matters more than the published 32K context limit.

H100, A100, 4× H200, and smaller B200 pods are intentionally not recommended here. The current upstream GLM-5.2 material does not provide comparable deployment evidence for them.

If a GPU configuration is not in the published recipe, don't assume it is equivalent just because the weights look as though they should fit. For a model this large, the runtime, low-precision kernels, KV-cache budget, parallelism strategy, and sustained load matter as much as raw HBM capacity.

Sources

Z.ai

• GLM-5.2 model card
  https://huggingface.co/zai-org/GLM-5.2

• GLM-5.2 FP8 checkpoint
  https://huggingface.co/zai-org/GLM-5.2-FP8

vLLM

• Official GLM-5.2 deployment recipe
  https://recipes.vllm.ai/zai-org/GLM-5.2

NVIDIA

• GLM-5.2 NVFP4 checkpoint
  https://huggingface.co/nvidia/GLM-5.2-NVFP4

AMD

• GLM-5.2 MXFP4 checkpoint
  https://huggingface.co/amd/GLM-5.2-MXFP4