The official open checkpoint is:

Qwen/Qwen3.8-2.4T-A95B

The model has 512 routed experts and selects 10 of them per token alongside one shared expert. Its 92-layer backbone mixes 69 Gated DeltaNet linear-attention layers with 23 full-attention layers, with full attention appearing every fourth layer.

Native context is 262,144 tokens, with an extended configuration available up to roughly 1.01 million tokens.

The open checkpoint is text-only and always uses reasoning. This is different from Qwen's hosted Qwen3.8-Max service, which adds features such as vision input and non-thinking mode.

For GPU deployment, the main decision is not whether 2.4T parameters will somehow fit. It is which precision format gives you a documented configuration on the hardware you actually have.

Start with the checkpoint that matches your GPUs

The practical options today are:

The full BF16 checkpoint is roughly 4.45 TiB. The official FP8 version is around 2.27 TiB, while the NVFP4 checkpoint used in the NVIDIA eight-GPU recipe is around 1.32 TiB.

That is why NVFP4 is the most approachable NVIDIA deployment if your goal is simply to get Qwen3.8 running without moving immediately to a 16-GPU cluster.

H100, H200, A100, B200 and smaller GPU configurations are not included here. Current vLLM material contains sizing information for some of those GPUs, but not equivalent end-to-end serving recipes. This guide does not turn memory estimates into deployment claims.

Before starting the model

The current vLLM recipe recommends a recent nightly build rather than an older stable release.

Create an environment with:

uv venv
source .venv/bin/activate

uv pip install -U vllm \
  --extra-index-url https://wheels.vllm.ai/nightly

uv pip install -U "transformers>=5.4.0"

Once you have a working build, pin it. Qwen3.8 relies on recently added model and kernel support, so continuously upgrading a production server to whatever nightly happens to be current is unnecessary risk.

Running Qwen3.8 on 8× B300 or GB300

For NVIDIA, the smallest documented configuration uses:

Inferact/Qwen3.8-2.4T-A95B-NVFP4

with eight GPUs.

Start with:

vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \
  --tensor-parallel-size 8 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

This configuration applies to an eight-GPU B300 system and is also documented across eight GB300 GPUs spanning two NVL4 trays.

vLLM's optimized NVFP4 path also uses:

--linear-backend flashinfer_cutedsl

but that flag depends on having the matching FlashInfer environment. Add it when reproducing the corresponding vLLM container/software stack rather than assuming every vLLM installation has the required backend.

Keep MTP enabled when you benchmark

Qwen3.8 contains a built-in Multi-Token Prediction head. vLLM's published results show that using three speculative tokens can make a substantial difference.

On its low-latency tests:

FP8 TP16
Without MTP: 130 output tok/s/user
MTP-3:       307 output tok/s/user

NVFP4 TP8
Without MTP: 133 output tok/s/user
MTP-3:       304 output tok/s/user

Those are measurements from vLLM's hardware and workload, not promised performance for every server. They do make MTP-3 worth testing from the beginning:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Using only one speculative token performed much less convincingly in the same testing. vLLM measured 64.8% acceptance for MTP-1, and at higher concurrency the extra speculative work could actually hurt throughput.

Running the official FP8 checkpoint

If you want Qwen's official FP8 checkpoint:

Qwen/Qwen3.8-2.4T-A95B-FP8

the documented low-latency configuration moves to 16 GPUs.

On B300, that normally means two eight-GPU nodes using TP16.

The head node runs:

vllm serve Qwen/Qwen3.8-2.4T-A95B-FP8 \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_ADDR \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The second node uses the same distributed configuration with:

--node-rank 1 \
--headless

Only rank 0 should expose the API server.

There is also a verified 12× GB300 FP8 configuration using:

TP4 × PP3

Why not TP12? Qwen3.8 has 64 full-attention heads, so its tensor-parallel size needs to divide 64. TP12 does not.

vLLM verified the TP4 × PP3 layout on 12 GB300 GPUs, including model loading, CUDA graph capture and generation. It is a useful option when you have three four-GPU GB300 trays, although the ordinary TP8 and TP16 setups remain simpler.

Running it on AMD

The documented AMD route uses:

Inferact/Qwen3.8-2.4T-A95B-MXFP4

on 8× MI355X.

vllm serve Inferact/Qwen3.8-2.4T-A95B-MXFP4 \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

One difference from the NVIDIA command is worth preserving: the current vLLM recipe does not recommend blindly adding FP8 KV cache to this ROCm setup.

Do not assume:

--kv-cache-dtype fp8

works with every ROCm/vLLM combination. Add it only after confirming it on the exact software build running on the node.

AMD also publishes:

amd/Qwen3.8-2.4T-A95B-Quark-MXFP4

for MI350 and MI355 hardware.

In that conversion, routed experts use OCP MXFP4 while components such as attention, routers, the shared expert, LM head and MTP layer remain at higher precision.

AMD's published GSM8K reproduction reported:

FP8 baseline: 97.49
MXFP4:        97.49

That result was reproduced with SGLang at TP8 on MI35x hardware. It is evidence for the quality of AMD's quantization, not a vLLM throughput benchmark.

How much context should you actually allocate?

The open checkpoint natively supports:

262,144 tokens

and can be extended to about:

1,010,000 tokens

with vLLM:

export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

and:

--max-model-len 1010000 \
--hf-overrides '{"max_position_embeddings":1010000}'

There is little reason to enable 1M context automatically.

Long context consumes memory that could otherwise be used for concurrent requests. vLLM's own NVFP4 testing illustrates the tradeoff: a configuration sized around the 262K window could hold roughly 25 concurrent requests in the available KV-cache budget, while a much shorter workload around 8K input plus 1K output allowed hundreds.

If you're serving repository-scale coding agents, very long context may be worth the cost. If most requests are 10K or 20K tokens, reserving for one million tokens mostly reduces how many users the GPUs can serve at once.

Set:

--max-model-len

for the workload you actually expect.

Reasoning and tool calling

Qwen3.8 is a reasoning model. The open checkpoint does not expose a normal non-thinking mode.

It supports three reasoning-effort settings:

xhigh
medium
low

xhigh is the default.

Qwen also preserves thinking across turns by default, which matters for agentic sessions where earlier reasoning and tool interactions are part of the ongoing context.

Start vLLM with:

--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder

Qwen's recommended generation settings include:

temperature = 1.0
top_p = 0.95
top_k = 20
min_p = 0.0
presence_penalty = 0.0
repetition_penalty = 1.0

A normal OpenAI-compatible client looks like:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
    timeout=3600,
)

response = client.chat.completions.create(
    model="Inferact/Qwen3.8-2.4T-A95B-NVFP4",
    messages=[
        {
            "role": "user",
            "content": "Review this distributed queue design and identify its failure modes."
        }
    ],
    reasoning_effort="medium",
    temperature=1.0,
    top_p=0.95,
    max_tokens=16384,
)

print(response.choices[0].message)

For hard coding and agent tasks, leave enough output budget for the reasoning trace. An aggressive max_tokens limit can terminate the generation before the model reaches its final answer.

Loading a model this large

At Qwen3.8's scale, startup time becomes part of operating the server.

vLLM's NVFP4 testing found that:

--load-format fastsafetensors \
--safetensors-load-strategy lazy

reduced model load time from 545 seconds to 306 seconds on the shared storage used in that test.

That result will vary with storage performance, but persistent model storage is clearly preferable to downloading or repeatedly copying a terabyte-scale checkpoint whenever a machine restarts.

vLLM also uses a larger engine startup timeout:

export VLLM_ENGINE_READY_TIMEOUT_S=3600

And don't make your own readiness decision solely from whether the process is alive. Send a small request to:

/v1/chat/completions

and verify that the model can actually generate.

Which GPU setup should you use?

If you're renting GPUs specifically for Qwen3.8-2.4T-A95B, the current documented choices are fairly straightforward.

8× B300 + NVFP4 is the simplest NVIDIA configuration in the current vLLM recipe.

8× GB300 + NVFP4 gives you the equivalent Grace Blackwell path across two NVL4 trays.

Use 16× B300 or GB300 + FP8 when you specifically want the official FP8 checkpoint and have enough hardware for TP16.

If you have three GB300 trays, 12× GB300 FP8 with TP4 × PP3 is an upstream-verified alternative.

On AMD, use 8× MI355X + MXFP4.

H100, H200, A100, B200 and smaller GPU counts are intentionally not recommended here. There may be enough aggregate memory in some of those configurations, but the current sources do not provide the same reproducible end-to-end deployment evidence.

For Qwen3.8, choosing the GPUs is only part of the deployment. The exact quant, tensor-parallel layout, KV-cache budget, vLLM build, network topology and MTP configuration all affect whether the resulting server is useful once real traffic starts hitting it.

Sources

Qwen

• Qwen3.8-2.4T-A95B official model card
  https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B

• Official FP8 checkpoint
  https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8

vLLM

• Official Qwen3.8-2.4T-A95B deployment recipe
  https://recipes.vllm.ai/Qwen/Qwen3.8-2.4T-A95B

• vLLM Qwen3.8 launch notes and benchmarks
  https://vllm.ai/blog/2026-08-12-qwen3.8

NVIDIA / Inferact

• Qwen3.8 NVFP4 checkpoint
  https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4

• Qwen3.8 MXFP4 checkpoint
  https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4

AMD

• AMD Qwen3.8 Quark MXFP4 checkpoint
  https://huggingface.co/amd/Qwen3.8-2.4T-A95B-Quark-MXFP4

The official checkpoint is:

moonshotai/Kimi-K3

K3 is unusual in that the released model is already designed around low-precision inference. Moonshot uses quantization-aware training, and the checkpoint ships with MXFP4 weights and MXFP8 activations. There is no separate "find a good 4-bit quant first" step before serving it.

Its attention stack combines Kimi Delta Attention, or KDA, with periodic full-attention layers. KDA keeps recurrent state rather than growing a conventional KV cache at every layer, which is part of what makes the million-token context window feasible.

If you're renting GPUs specifically to run Kimi K3, the hardware choices are fairly narrow. This is not a model where adding up VRAM across a few H100s is enough to decide whether a pod will work.

Start with the native Moonshot checkpoint

There are a few K3 variants, but the native release has the cleanest deployment path:

Red Hat's NVFP4 conversion is intended for vLLM, but its model card still carries compatibility notes around upstream support.

AMD's Quark checkpoint is vendor-published and has useful accuracy results, but its current reproduction path requires a small vLLM modification. Unless that particular quant is the reason you're deploying K3, bring up the native checkpoint first.

The vLLM environment

The current K3 recipe uses vLLM 0.27.1 or newer and provides dedicated images rather than expecting you to assemble all of the model-specific kernels yourself.

For NVIDIA:

vllm/vllm-openai:kimi-k3

For AMD:

vllm/vllm-openai_rocm:kimi-k3

The NVIDIA image currently uses CUDA 13, so the host needs an R580-series or newer NVIDIA driver.

With a model this large, using the dedicated image is worth it. K3 depends on several specialized attention, MoE, loading, and communication paths, and reproducing those from an arbitrary vLLM install adds failure points before the model even starts loading.

Which GPUs can you actually use?

The current upstream deployment paths are:

H200, H100, A100, and 8× B200 are deliberately missing from that list.

There is an open vLLM Recipes issue around the published H200 path involving an OOM followed by a context-parallel runtime error. That does not prove K3 is impossible on H200; it does mean there is not a clean upstream recipe worth relying on today.

Likewise, the documented B200 configuration is 16 GPUs, not eight. A large aggregate VRAM number is not enough reason to turn an 8× B200 pod into a K3 recommendation.

Running K3 on 8× B300 or GB300

For NVIDIA, this is the simplest place to begin.

The official vLLM quick-start is:

vllm serve moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

fastsafetensors matters more than usual here because the K3 repository is enormous. Model-loading speed becomes part of the deployment experience.

Prefix caching is also worth enabling deliberately:

--enable-prefix-caching

K3 has a hybrid cache: recurrent KDA state in most layers and conventional KV cache in the full-attention layers. vLLM knows how to handle that hybrid structure, but prefix caching is not something to assume is active unless it has been enabled.

If you want FP8 KV cache, use:

--kv-cache-dtype fp8

with the attention configuration from the K3 recipe:

--attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}'

A fuller server command becomes:

vllm serve moonshotai/Kimi-K3 \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name kimi-k3 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --kv-cache-dtype fp8 \
  --attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

On GB200 and GB300 NVLink systems, the current recipe also recommends:

export NCCL_MNNVL_ENABLE=1
export NCCL_CUMEM_ENABLE=1
export NCCL_NVLS_ENABLE=1

Those variables belong to the NVLink setup; they are not generic flags to copy onto unrelated PCIe hardware.

If you have B200, plan for 16 GPUs

K3's documented B200 deployment uses:

16× B200
TP8 × PP2

So the model is split with tensor parallelism across eight GPUs and pipeline parallelism across two stages.

The serving configuration includes:

vllm serve moonshotai/Kimi-K3 \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name kimi-k3 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --kv-cache-dtype fp8 \
  --attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

That is only the model-side configuration. A 16-GPU deployment still needs the nodes, ranks, and network interfaces configured correctly for distributed vLLM.

If your provider only offers an eight-GPU B200 pod, do not assume that the same command with --pipeline-parallel-size 1 gives you an equivalent supported setup. It does not match the published topology.

Running K3 on AMD

The clean AMD route uses 8× MI355X or MI350X and the native Moonshot checkpoint.

A starting MI355X configuration is:

export VLLM_ROCM_USE_AITER=1
export SAFETENSORS_FAST_GPU=1
export VLLM_USE_BREAKABLE_CUDAGRAPH=0

vllm serve moonshotai/Kimi-K3 \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name kimi-k3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --load-format auto \
  --gpu-memory-utilization 0.95 \
  --mm-encoder-tp-mode data \
  --max-num-seqs 128 \
  --max-num-batched-tokens 4096 \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

The vision encoder is handled differently from the language model because it is tiny by comparison and does not benefit from being split in the same way across TP8.

AMD also publishes:

amd/Kimi-K3-Quark-MXFP4-AttnFP8

for MI350 and MI355. It stores the experts and shared experts in OCP MXFP4 while keeping attention in FP8.

AMD reports 96.36 on 5-shot GSM8K versus 96.82 for the native model in its published evaluation.

The catch is that the current reproduction instructions ask you to modify vLLM's quark_ocp_mx.py and set:

self.emulate = True

That is manageable if you're building your own image, but it is one more moving part. The native K3 checkpoint is the simpler deployment if you just want the model running.

DSpark can make decoding much faster

K3 has an open DSpark draft model:

Inferact/Kimi-K3-DSpark

Add it to a compatible deployment with:

--speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

vLLM's GB300 testing reported:

TP16 without DSpark: 118 tok/s
TP16 with DSpark:    370 tok/s

That is about a 3.1× increase in single-stream decode speed in that specific test.

It is worth testing if decode latency matters, but don't budget a deployment around "3× faster" before measuring your own prompts. Speculative decoding depends on how often the draft model's tokens are accepted.

The million-token context is real, but expensive

K3 supports:

1,048,576 tokens

natively.

Most of its layers use KDA recurrent state rather than a conventional ever-growing KV cache, while periodic full-attention layers preserve global recall. This gives K3 a much better long-context memory profile than a model where every layer uses ordinary full attention.

It still does not make a million-token request cheap.

Long context competes with concurrent users, output tokens, model buffers, attention state, and everything else living in GPU memory. A K3 pod that works well for 32K or 128K requests does not automatically have useful concurrency at 1M.

Set the context limit around what you're actually doing with the model. If the endpoint is for codebase-scale agents, long context may be worth the trade. If most requests are relatively short, giving every request access to a million tokens mostly consumes capacity that could have served other requests.

Reasoning and tool calls need one K3-specific detail

K3 exposes its reasoning in:

reasoning_content

and supports:

low
high
max

for reasoning_effort.

max is the default.

vLLM needs:

--reasoning-parser kimi_k3 \
--tool-call-parser kimi_k3 \
--enable-auto-tool-choice

There is one behavior that matters a lot if you're building agents: K3 was trained with preserved thinking history.

When the assistant has already made a tool call or completed a reasoning turn, send the complete previous assistant message back on the next turn, including:

reasoning_content
content
tool_calls

Saving only the visible content throws away information the model expects to see in a multi-turn agent session.

Tool-call output should also be schema-validated before execution. vLLM notes that K3 can occasionally emit a tool-call shape that its parser does not accept cleanly. If the generated call is malformed, reject or retry it rather than sending malformed arguments into a real tool.

K3 can take images too

The native checkpoint is multimodal, so the same server can accept text and image input through the OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
    timeout=3600,
)

response = client.chat.completions.create(
    model="kimi-k3",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/image.png"
                    }
                },
                {
                    "type": "text",
                    "text": "Explain what is happening in this image."
                }
            ]
        }
    ],
    max_tokens=2048,
)

print(response.choices[0].message.content)

That makes one K3 deployment useful for coding agents that also need to inspect screenshots, diagrams, UI states, or other visual input.

When 8 GPUs are no longer enough

For larger shared services, NVIDIA Dynamo publishes K3 deployments on GB200 and GB300 NVL72 systems:

16× GB200    aggregated TP16
32× GB200    16 prefill + 16 decode

16× GB300    two TP8 replicas
24× GB300    8 prefill + 16 decode

Those are not just bigger versions of the vllm serve command above. Dynamo adds routing, distributed serving, and KV transfer between workers.

If you're bringing up one K3 endpoint, start with 8× B300/GB300 or 8× MI355X. Dynamo starts making sense when the problem changes from "run the model" to "serve substantial shared traffic."

Which K3 setup should you choose?

If you're selecting GPUs for K3 today, the simplest documented choices are:

NVIDIA: use 8× B300 or 8× GB300 with the native Moonshot checkpoint.

B200: the published route is 16× B200, using TP8 × PP2.

AMD: use 8× MI355X or MI350X with the native checkpoint.

The alternative NVFP4 and Quark checkpoints are worth testing when you have a specific reason to optimize around those formats. DSpark is worth benchmarking once the baseline server is stable.

H200, H100, A100, and 8× B200 should not be treated as equivalent alternatives just because their aggregate memory looks plausible. The current upstream deployment evidence is not at the same level.

For Kimi K3, a useful GPU setup is one that can reproduce the model load, context length, reasoning parser, tool calls, multimodal input, concurrency, and sustained generation on a pinned runtime. That's the standard that matters once you're paying for the GPUs.

Sources

Moonshot AI

• Kimi K3 official model card
  https://huggingface.co/moonshotai/Kimi-K3

• Moonshot AI
  https://www.moonshot.ai/

vLLM

• Official Kimi K3 deployment recipe
  https://recipes.vllm.ai/moonshotai/Kimi-K3

• Kimi K3 launch guide and benchmarks
  https://vllm.ai/blog/2026-07-27-k3

• Current H200 deployment issue
  https://github.com/vllm-project/recipes/issues/698

NVIDIA

• NVIDIA Dynamo Kimi K3 production recipes
  https://docs.nvidia.com/dynamo/dev/recipes/kimi-k3

AMD

• AMD Kimi K3 Quark MXFP4 + FP8 checkpoint
  https://huggingface.co/amd/Kimi-K3-Quark-MXFP4-AttnFP8

Alternative Quantization

• Red Hat Kimi K3 NVFP4
  https://huggingface.co/RedHatAI/Kimi-K3-NVFP4

If you are renting GPUs to run it yourself, two parts of the architecture matter more than the raw parameter count.

The first is IndexShare. Instead of running a separate sparse-attention indexer in every layer, GLM-5.2 reuses one indexer across groups of four layers. Z.ai reports that this cuts per-token FLOPs by about 2.9× at 1M context.

The second is Multi-Token Prediction, or MTP. GLM-5.2 can draft up to five tokens ahead during decoding. Z.ai reports roughly a 20% increase in accepted draft length over the previous GLM generation. Whether that produces a worthwhile speedup on your server still depends on how often those draft tokens are accepted on your actual prompts.

For most vLLM deployments, start with:

zai-org/GLM-5.2-FP8

The FP8 checkpoint is much easier to deploy than the full BF16 model and has the broadest set of published GPU recipes.

Which checkpoint fits your hardware?

The main choices are:

If you simply want to get GLM-5.2 running, FP8 is the least complicated starting point.

The full BF16 checkpoint needs a larger multi-node configuration and extra loader setup, so there is little reason to begin there unless BF16 is specifically required.

NVIDIA’s NVFP4 checkpoint is interesting on Blackwell because it reduces the model footprint substantially. AMD has a separate MXFP4 checkpoint for MI355X.

Set up vLLM before renting a long-lived pod

The current GLM-5.2 recipe uses:

vLLM 0.23.0

as its stable baseline.

Install it with:

uv venv
source .venv/bin/activate

uv pip install "vllm==0.23.0" --torch-backend=auto
uv pip install "transformers>=5.9.0"

There are also dedicated Docker images:

vllm/vllm-openai:glm52

or, for CUDA 12.x:

vllm/vllm-openai:glm52-cu129

A basic launch looks like:

docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:glm52 \
  zai-org/GLM-5.2-FP8 \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name glm-5.2-fp8 \
  --max-model-len 131072

For good FP8 performance, the current recipe also expects DeepGEMM. Install it using the install_deepgemm.sh helper supplied by the vLLM setup.

One version caveat is worth knowing before debugging speculative decoding: if you need tool calling and MTP at the same time, the current vLLM guidance points to the latest main branch because not every relevant fix is present in 0.23.0.

Which GPUs should you rent?

These are the useful upstream-backed configurations:

H100, A100, 4× H200, and smaller B200 configurations are deliberately not included here. They may be possible, but the current GLM-5.2 recipe does not provide an equivalent end-to-end deployment path for them.

That distinction matters if you are paying by the hour. A configuration that looks plausible from VRAM alone is not the same thing as one with a published runtime, kernel path, and serving command.

8× H200 is the straightforward NVIDIA setup

For normal FP8 serving on Hopper, the published recipe uses TP8, FP8 KV cache, and GLM-5.2’s five-token MTP head:

vllm serve zai-org/GLM-5.2-FP8 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name glm-5.2-fp8

If you want a conventional NVIDIA pod and do not need the full million-token context window, this is the cleanest place to start.

H20 has a separate DSpark path

On 8× H20, vLLM publishes a recipe using an external DSpark speculator rather than the built-in MTP head:

vllm serve zai-org/GLM-5.2-FP8 \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --max-model-len 200000 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 32768 \
  --speculative-config '{"model":"RedHatAI/GLM-5.2-speculator.dspark","num_speculative_tokens":7,"method":"dspark"}' \
  --served-model-name glm-5.2-fp8

vLLM reports 70+ tokens/sec for a single request on this setup.

Treat that as a result from their exact configuration, not as the number every H20 rental will produce.

If you actually need 1M context, use the B200 recipe

The important full-context NVIDIA setup is:

8× B200
GLM-5.2-FP8
FP8 KV cache

Run:

VLLM_DEEP_GEMM_WARMUP=skip \
vllm serve zai-org/GLM-5.2-FP8 \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --max-num-seqs 32 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name glm-5.2-fp8

The setting to pay attention to here is:

--max-num-seqs 32

Every active request competes for the same KV-cache budget. If long-context requests start running out of memory, lower this value. If your prompts are shorter, you can usually afford more concurrency.

FP8 KV cache is also a major part of this configuration:

--kv-cache-dtype fp8_e4m3

It cuts the cache footprint enough to make the million-token window practical on the published B200 setup.

VLLM_DEEP_GEMM_WARMUP=skip reduces startup time by skipping the initial DeepGEMM JIT warmup. Kernels then compile as requests begin arriving.

Blackwell users can trade FP8 for NVFP4

NVIDIA publishes:

nvidia/GLM-5.2-NVFP4

for B200 and B300.

This is an NVIDIA Model Optimizer conversion, not a random community quant. The MoE expert linear layers move to NVFP4, while components such as the shared expert, attention, embeddings, and other parts of the model remain at higher precision.

Serve it with:

vllm serve nvidia/GLM-5.2-NVFP4 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --trust-remote-code \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --kv-cache-dtype fp8_e4m3 \
  --host 0.0.0.0 \
  --port 8000

You do not need:

--quantization nvfp4

because vLLM reads the quantization configuration from the checkpoint itself.

If you are choosing between FP8 and NVFP4 on Blackwell, NVFP4 is the option to test when model footprint matters more than sticking to the native FP8 release.

Running GLM-5.2 on AMD

For MI300X and MI355X, the FP8 route uses AMD’s AITER kernels:

VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
vllm serve zai-org/GLM-5.2-FP8 \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --gpu-memory-utilization 0.80 \
  --max-model-len 524288 \
  --max-num-seqs 32 \
  --linear-backend aiter \
  --moe-backend aiter

The published starting context here is 524,288 tokens. That is not the model’s maximum; GLM-5.2 still supports 1M. Increase it only after confirming that the server has enough KV-cache headroom.

The 0.80 GPU-memory utilization value is also intentional. It leaves space for ROCm runtime allocations and MTP graph capture instead of trying to occupy virtually every byte of HBM on startup.

MI355X also has an MXFP4 option

AMD publishes:

amd/GLM-5.2-MXFP4

and the current recipe uses 8× MI355X with TP8:

export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_USE_AITER_FP8BMM=0
export VLLM_ROCM_USE_AITER_FP4BMM=0

vllm serve amd/GLM-5.2-MXFP4 \
  --trust-remote-code \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --no-enable-prefix-caching \
  --served-model-name glm-5.2-mxfp4

There is a significant tradeoff here: the published MXFP4 recipe uses only a 32K context window.

If you are choosing GLM-5.2 specifically because you need hundreds of thousands of tokens of context, use the FP8 ROCm recipe instead.

AMD reports 93.93 on GSM8K versus 94.09 for the base model, or roughly 99.8% score recovery, for this MXFP4 conversion.

The MTP layer itself remains at higher precision. The published command simply does not enable speculative decoding, so do not read the absence of --speculative-config as evidence that MTP has been removed from the checkpoint.

Don't allocate 1M context just because the model supports it

GLM-5.2 can handle:

1,048,576 tokens

but there is no reason for every endpoint to reserve enough memory for million-token requests.

The current recipes themselves use very different context limits:

A coding agent working over a giant repository may justify 500K or 1M context. A normal API dominated by 10K prompts probably does not.

Longer context consumes KV cache that could otherwise serve more users, so choose --max-model-len around the traffic you expect rather than the largest number on the model card.

Reasoning and tool calling

Thinking is enabled by default.

GLM-5.2 supports three useful modes:

Think Max
Think High
Non-thinking

Think Max is the default.

For Think High:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5.2-fp8",
    "messages": [
      {
        "role": "user",
        "content": "Design a fault-tolerant distributed queue."
      }
    ],
    "temperature": 1,
    "max_tokens": 4096,
    "chat_template_kwargs": {
      "reasoning_effort": "high"
    }
  }'

For a fast non-thinking response, pass:

{
  "enable_thinking": false
}

through chat_template_kwargs.

If you are building an agent or tool-using application, start vLLM with:

--tool-call-parser glm47 \
--reasoning-parser glm45 \
--enable-auto-tool-choice

Which GLM-5.2 setup should you choose?

If you are renting GPUs specifically for GLM-5.2, the current upstream paths make the decision fairly simple.

8× H200 + FP8 is the straightforward NVIDIA setup.

8× H20 + FP8 is the documented choice if you specifically want the DSpark serving path.

If you really need the full 1M context window, use 8× B200 + FP8.

On Blackwell, 8× B200 or B300 + NVIDIA NVFP4 is the lower-footprint alternative.

For AMD, use 8× MI300X or MI355X + FP8 when long context matters. Use 8× MI355X + MXFP4 when reducing the model footprint matters more than the published 32K context limit.

H100, A100, 4× H200, and smaller B200 pods are intentionally not recommended here. The current upstream GLM-5.2 material does not provide comparable deployment evidence for them.

If a GPU configuration is not in the published recipe, don't assume it is equivalent just because the weights look as though they should fit. For a model this large, the runtime, low-precision kernels, KV-cache budget, parallelism strategy, and sustained load matter as much as raw HBM capacity.

Sources

Z.ai

• GLM-5.2 model card
  https://huggingface.co/zai-org/GLM-5.2

• GLM-5.2 FP8 checkpoint
  https://huggingface.co/zai-org/GLM-5.2-FP8

vLLM

• Official GLM-5.2 deployment recipe
  https://recipes.vllm.ai/zai-org/GLM-5.2

NVIDIA

• GLM-5.2 NVFP4 checkpoint
  https://huggingface.co/nvidia/GLM-5.2-NVFP4

AMD

• GLM-5.2 MXFP4 checkpoint
  https://huggingface.co/amd/GLM-5.2-MXFP4

deepseek-ai/DeepSeek-V4-Flash-0731

It keeps the same underlying architecture as the earlier V4 Flash preview, but DeepSeek re-trained the model for much stronger agentic performance. The published results are a big jump: 82.7 on Terminal Bench 2.1 versus 61.8 for the preview, and 54.4 on DeepSWE versus 7.3.

The model is MIT-licensed and ungated.

Under the hood, V4 Flash is a 284B-parameter Mixture-of-Experts model with roughly 13B parameters active per token. It supports a native context window of 1,048,576 tokens and uses a hybrid attention design combining Compressed Sparse Attention (CSA) with Heavily Compressed Attention (HCA).

The checkpoint is already aggressively compressed. Routed experts are stored in MXFP4 with FP8 scales, while linear, attention, and router layers use FP8. At serving time, the KV cache can also be stored in FP8.

DeepSeek reports that V4 needs roughly 27% of V3.2's inference FLOPs per token and about 10% of its KV-cache footprint at 1M context.

If Hugging Face shows roughly 304–305B parameters for the repository, that does not contradict the 284B model size. The fused 0731 repository also contains the DSpark speculative-decoding module.

The checkpoint is around 167 GB on disk, compared with roughly 160 GB for the preview. Do not use those figures as direct VRAM requirements. vLLM's MI325X validation, for example, measured about 148.66 GiB of resident model memory for the preview checkpoint.

That difference is partly units: Hugging Face reports decimal GB, while GPU memory is normally discussed in GiB. More importantly, loading the weights is only part of the memory budget. KV cache, runtime buffers, kernels, and concurrency all need room too.

Which checkpoint should you use?

For most users, start with:

deepseek-ai/DeepSeek-V4-Flash-0731

The other variants are useful when a particular GPU or runtime path requires them:

One detail matters if you are choosing NVIDIA's NVFP4 checkpoint: it is based on the preview weights, not 0731. You get the NVIDIA NVFP4 execution path, but not the newer post-training that produced the large agentic gains above.

The community W4A16 conversion is also not proof that stock 0731 runs cleanly on A100. Its published end-to-end validation used Radeon Pro W6800X Duo hardware with CPU offload.

Pin vLLM before you start

The minimum runtime depends on the path you are using:

For the documented NVIDIA 0731 setup:

docker pull vllm/vllm-openai:v0.25.0

Once you have a working deployment, keep the version pinned. A model this dependent on specialized kernels is not a good place to discover that latest changed underneath you.

Which GPUs can you actually use?

The current DeepSeek and vLLM recipes give you these reproducible paths:

For ordinary multi-GPU serving on H200, B200, and B300, vLLM's recommended pattern is:

Data Parallel = 4
Expert Parallel = enabled

H100, A100, 2× H200, 2× B200, 1× B200, and MI300X are not included here because the current recipe does not provide the same level of end-to-end deployment evidence.

4× GB300: the cleanest 0731 reference

DeepSeek publishes this command for a single 4× GB300 node:

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

This is specifically a Blackwell recipe.

These two options:

--moe-backend deep_gemm_mega_moe
--attention-config '{"use_fp4_indexer_cache": true}'

should not be copied onto H200.

You may also see the FP4 indexer setting written upstream as:

--attention_config.use_fp4_indexer_cache True

Use the syntax supported by the vLLM build you pinned.

B200 and B300

For 4× B200 or 4× B300, a useful starting point is:

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice

To add DSpark:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

There is a small upstream disagreement here: DeepSeek's model card uses greedy, while the current vLLM recipe generator uses probabilistic.

If speculative decoding matters to your workload, benchmark both rather than treating either value as universally best.

H200 uses a different path

H200 is Hopper, so the Blackwell FP4 indexer and deep_gemm_mega_moe overrides are intentionally absent.

Start with:

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --max-model-len auto

If you have a full 8× H200 node, vLLM also documents a more advanced layout with four GPUs handling prefill and four handling decode. KV state is transferred using Mooncake or NIXL, with vllm-router in front.

Those published prefill/decode examples currently use the preview checkpoint, not 0731, so keep that distinction if you are reproducing them exactly.

RTX PRO 6000 Blackwell needs two extra setup steps

vLLM has an explicit 0731 validation for:

8× RTX PRO 6000 Blackwell
TP8
Expert Parallel

using:

vllm/vllm-openai:v0.25.0

Before starting the model, install the required FlashInfer build:

/usr/bin/python3.12 -m pip install \
  --no-deps \
  flashinfer-python==0.6.14

export FLASHINFER_DISABLE_VERSION_CHECK=1

The --no-deps is important. Installing FlashInfer normally can replace PyTorch and break vLLM's compiled extensions.

Then run:

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4

On this particular profile, do not enable MTP or DSpark. Both currently fail during warmup on the sm_120 sparse-MLA path.

AMD: one MI325X can run it, with limits

The single-GPU MI325X validation is one of the more useful upstream tests because vLLM publishes actual memory observations rather than just saying the model fits.

The validated setup is:

1× MI325X 256GB
TP1
4K context
10 GB explicit KV cache

Run:

export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --host 0.0.0.0 \
  --port 8002 \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 4096 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 256 \
  --kv-cache-memory-bytes 10000000000 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --tokenizer-mode deepseek_v4 \
  --moe-backend triton_unfused \
  --enforce-eager

vLLM measured about 148.66 GiB of model memory after loading and reports successful chat, non-thinking, and Think High tests over a 24-hour allocation.

This is a useful single-GPU option if 4K context is enough. It should not be interpreted as a one-GPU million-token deployment.

The published AMD validation uses the preview checkpoint. If you want 0731 with DSpark on ROCm, use vLLM 0.26.0 or newer.

4× MI355X

The other documented AMD path uses TP4 across four MI355X GPUs:

export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --host localhost \
  --port 8001 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 4 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --compilation-config '{"mode":3,"cudagraph_mode":"FULL_DECODE_ONLY"}'

The published validation also includes 8-shot GSM8K results:

Flexible-extract exact match: 0.9439
Strict-match exact match:     0.9431

TP8 is available when lower latency matters more than reproducing the published TP4 setup.

DGX Spark is a special case

A single DGX Spark GB10 has 128 GB of unified memory, so the documented path uses two machines at TP2, one GB10 per node.

It also uses a specialized image:

eugr/spark-vllm-b12x:latest

rather than the standard vLLM container, plus ConnectX/RoCE network configuration between the nodes.

If you are using DGX Spark, follow that recipe as its own deployment path. It is not equivalent to a normal two-GPU Blackwell pod.

vLLM also carries a single-GPU DGX Station GB300 profile. The currently published command uses the preview checkpoint with MTP rather than the 0731 DSpark release.

Don't size every server for 1M context

DeepSeek V4 Flash supports:

1,048,576 tokens

but that does not mean every deployment should reserve enough memory to serve a million-token request.

The MI325X profile makes the distinction obvious: the model supports 1M context, while that particular validated server configuration supports 4K.

Several vLLM recipes therefore use:

--max-model-len auto

instead of forcing the model's architectural maximum.

V4 also has a hybrid CSA+HCA cache, so generic dense-transformer calculations such as "X GB per 100K tokens" are not reliable across every configuration.

Use the memory profile vLLM prints at startup. If you need to reserve a fixed cache budget, use:

--kv-cache-memory-bytes

Most published configurations use:

--kv-cache-dtype fp8

or, on ROCm:

--kv-cache-dtype fp8_e4m3

to reduce cache memory.

If you are using --data-parallel-size 4, remember that scheduling settings such as --max-num-seqs and --max-num-batched-tokens apply per data-parallel rank, not once for the entire four-GPU server.

Reasoning and tool calling

DeepSeek V4 Flash supports three reasoning levels:

low
high
max

DeepSeek's recommended sampling depends on what you are doing:

If you are serving Think High or Think Max, leave room for very long outputs. DeepSeek recommends allowing up to 384K output tokens for the deeper reasoning modes, and the vLLM recipe notes that Think Max needs:

--max-model-len >= 393216

to avoid truncating the reasoning budget.

For the OpenAI-compatible vLLM endpoint, start the server with:

--tokenizer-mode deepseek_v4 \
--reasoning-parser deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice

A normal request looks like:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Flash-0731",
    "messages": [
      {
        "role": "user",
        "content": "Explain optimistic and pessimistic locking."
      }
    ],
    "temperature": 1.0,
    "top_p": 1.0,
    "max_tokens": 2048
  }'

For an agentic request with Think High:

from openai import OpenAI

client = OpenAI(
    base_url="http://YOUR_SERVER_IP:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash-0731",
    messages=[
        {
            "role": "user",
            "content": "Find the race condition in this worker design."
        }
    ],
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "high"
        }
    },
)

print(response.choices[0].message)

Keep the model on persistent storage

Current Hugging Face tooling uses the hf CLI:

export HF_HOME=/workspace/huggingface
mkdir -p "$HF_HOME"

hf download deepseek-ai/DeepSeek-V4-Flash-0731

At roughly 167 GB, repeatedly downloading the checkpoint every time a pod is recreated gets expensive in both time and bandwidth. Persistent model storage is worth setting up from the beginning.

Which DeepSeek V4 Flash setup should you choose?

If you are choosing GPUs specifically for DeepSeek-V4-Flash-0731, the current upstream paths are fairly clear.

4× GB300 is the cleanest reference because DeepSeek publishes the exact 0731 command.

4× B200 or B300 gives you the equivalent Blackwell DP4 + expert-parallel route.

4× H200 is the documented Hopper setup. Do not copy the Blackwell FP4-specific flags into it.

If you have 8× H200, vLLM also has a more advanced prefill/decode design, although the published commands currently use the preview checkpoint.

8× RTX PRO 6000 Blackwell is explicitly validated for 0731, but MTP and DSpark need to stay off in the current profile.

On AMD, 1× MI325X is a real validated option if a 4K context limit is acceptable, while 4× MI355X gives you the stronger multi-GPU ROCm path.

H100, A100, 2× H200, 2× B200, 1× B200, and MI300X are intentionally not recommended here. The current upstream material does not provide equivalent deployment recipes for them.

With V4 Flash, the useful question is not simply whether the checkpoint can fit in aggregate memory. You need enough headroom for the actual runtime, a supported kernel path for the GPU architecture, enough KV cache for the context you plan to serve, and a configuration that stays up under real concurrent traffic.

Sources

DeepSeek

• DeepSeek V4 Flash 0731 model card
  https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731

• DeepSeek V4 Flash preview — architecture and precision
  https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash

• DeepSeek V4 technical report
  https://arxiv.org/abs/2606.19348

vLLM

• DeepSeek V4 Flash deployment recipe
  https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash

• Raw hardware profiles and validations
  https://raw.githubusercontent.com/vllm-project/recipes/main/models/deepseek-ai/DeepSeek-V4-Flash.yaml

• FP8 KV-cache documentation
  https://docs.vllm.ai/en/latest/features/quantization/quantized\_kvcache/

Alternative checkpoints

• NVIDIA DeepSeek V4 Flash NVFP4
  https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4

• Community 0731 NVFP4 conversion
  https://huggingface.co/MJPansa/DeepSeek-V4-Flash-0731-NVFP4

• Community 0731 INT4/W4A16 conversion
  https://huggingface.co/yiminyuan/DeepSeek-V4-Flash-0731-INT4-W4A16

• Unsloth DeepSeek V4 GGUF documentation
  https://unsloth.ai/docs/models/deepseek-v4