Deploying Kimi K3 with vLLM: Verified GPU Pods, Quants, and Serving Recipes

Kimi K3 is Moonshot AI's 2.8-trillion-parameter Mixture-of-Experts model for coding, reasoning, agents, and multimodal work. It activates 16 of 896 routed experts for each token and supports a native context window of 1,048,576 tokens.

The official checkpoint is:

moonshotai/Kimi-K3

K3 is unusual in that the released model is already designed around low-precision inference. Moonshot uses quantization-aware training, and the checkpoint ships with MXFP4 weights and MXFP8 activations. There is no separate "find a good 4-bit quant first" step before serving it.

Its attention stack combines Kimi Delta Attention, or KDA, with periodic full-attention layers. KDA keeps recurrent state rather than growing a conventional KV cache at every layer, which is part of what makes the million-token context window feasible.

If you're renting GPUs specifically to run Kimi K3, the hardware choices are fairly narrow. This is not a model where adding up VRAM across a few H100s is enough to decide whether a pod will work.

Start with the native Moonshot checkpoint

There are a few K3 variants, but the native release has the cleanest deployment path:

Red Hat's NVFP4 conversion is intended for vLLM, but its model card still carries compatibility notes around upstream support.

AMD's Quark checkpoint is vendor-published and has useful accuracy results, but its current reproduction path requires a small vLLM modification. Unless that particular quant is the reason you're deploying K3, bring up the native checkpoint first.

The vLLM environment

The current K3 recipe uses vLLM 0.27.1 or newer and provides dedicated images rather than expecting you to assemble all of the model-specific kernels yourself.

For NVIDIA:

vllm/vllm-openai:kimi-k3

For AMD:

vllm/vllm-openai_rocm:kimi-k3

The NVIDIA image currently uses CUDA 13, so the host needs an R580-series or newer NVIDIA driver.

With a model this large, using the dedicated image is worth it. K3 depends on several specialized attention, MoE, loading, and communication paths, and reproducing those from an arbitrary vLLM install adds failure points before the model even starts loading.

Which GPUs can you actually use?

The current upstream deployment paths are:

H200, H100, A100, and 8× B200 are deliberately missing from that list.

There is an open vLLM Recipes issue around the published H200 path involving an OOM followed by a context-parallel runtime error. That does not prove K3 is impossible on H200; it does mean there is not a clean upstream recipe worth relying on today.

Likewise, the documented B200 configuration is 16 GPUs, not eight. A large aggregate VRAM number is not enough reason to turn an 8× B200 pod into a K3 recommendation.

Running K3 on 8× B300 or GB300

For NVIDIA, this is the simplest place to begin.

The official vLLM quick-start is:

vllm serve moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

fastsafetensors matters more than usual here because the K3 repository is enormous. Model-loading speed becomes part of the deployment experience.

Prefix caching is also worth enabling deliberately:

--enable-prefix-caching

K3 has a hybrid cache: recurrent KDA state in most layers and conventional KV cache in the full-attention layers. vLLM knows how to handle that hybrid structure, but prefix caching is not something to assume is active unless it has been enabled.

If you want FP8 KV cache, use:

--kv-cache-dtype fp8

with the attention configuration from the K3 recipe:

--attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}'

A fuller server command becomes:

vllm serve moonshotai/Kimi-K3 \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name kimi-k3 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --kv-cache-dtype fp8 \
  --attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

On GB200 and GB300 NVLink systems, the current recipe also recommends:

export NCCL_MNNVL_ENABLE=1
export NCCL_CUMEM_ENABLE=1
export NCCL_NVLS_ENABLE=1

Those variables belong to the NVLink setup; they are not generic flags to copy onto unrelated PCIe hardware.

If you have B200, plan for 16 GPUs

K3's documented B200 deployment uses:

16× B200
TP8 × PP2

So the model is split with tensor parallelism across eight GPUs and pipeline parallelism across two stages.

The serving configuration includes:

vllm serve moonshotai/Kimi-K3 \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name kimi-k3 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --kv-cache-dtype fp8 \
  --attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

That is only the model-side configuration. A 16-GPU deployment still needs the nodes, ranks, and network interfaces configured correctly for distributed vLLM.

If your provider only offers an eight-GPU B200 pod, do not assume that the same command with --pipeline-parallel-size 1 gives you an equivalent supported setup. It does not match the published topology.

Running K3 on AMD

The clean AMD route uses 8× MI355X or MI350X and the native Moonshot checkpoint.

A starting MI355X configuration is:

export VLLM_ROCM_USE_AITER=1
export SAFETENSORS_FAST_GPU=1
export VLLM_USE_BREAKABLE_CUDAGRAPH=0

vllm serve moonshotai/Kimi-K3 \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name kimi-k3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --load-format auto \
  --gpu-memory-utilization 0.95 \
  --mm-encoder-tp-mode data \
  --max-num-seqs 128 \
  --max-num-batched-tokens 4096 \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

The vision encoder is handled differently from the language model because it is tiny by comparison and does not benefit from being split in the same way across TP8.

AMD also publishes:

amd/Kimi-K3-Quark-MXFP4-AttnFP8

for MI350 and MI355. It stores the experts and shared experts in OCP MXFP4 while keeping attention in FP8.

AMD reports 96.36 on 5-shot GSM8K versus 96.82 for the native model in its published evaluation.

The catch is that the current reproduction instructions ask you to modify vLLM's quark_ocp_mx.py and set:

self.emulate = True

That is manageable if you're building your own image, but it is one more moving part. The native K3 checkpoint is the simpler deployment if you just want the model running.

DSpark can make decoding much faster

K3 has an open DSpark draft model:

Inferact/Kimi-K3-DSpark

Add it to a compatible deployment with:

--speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

vLLM's GB300 testing reported:

TP16 without DSpark: 118 tok/s
TP16 with DSpark:    370 tok/s

That is about a 3.1× increase in single-stream decode speed in that specific test.

It is worth testing if decode latency matters, but don't budget a deployment around "3× faster" before measuring your own prompts. Speculative decoding depends on how often the draft model's tokens are accepted.

The million-token context is real, but expensive

K3 supports:

1,048,576 tokens

natively.

Most of its layers use KDA recurrent state rather than a conventional ever-growing KV cache, while periodic full-attention layers preserve global recall. This gives K3 a much better long-context memory profile than a model where every layer uses ordinary full attention.

It still does not make a million-token request cheap.

Long context competes with concurrent users, output tokens, model buffers, attention state, and everything else living in GPU memory. A K3 pod that works well for 32K or 128K requests does not automatically have useful concurrency at 1M.

Set the context limit around what you're actually doing with the model. If the endpoint is for codebase-scale agents, long context may be worth the trade. If most requests are relatively short, giving every request access to a million tokens mostly consumes capacity that could have served other requests.

Reasoning and tool calls need one K3-specific detail

K3 exposes its reasoning in:

reasoning_content

and supports:

low
high
max

for reasoning_effort.

max is the default.

vLLM needs:

--reasoning-parser kimi_k3 \
--tool-call-parser kimi_k3 \
--enable-auto-tool-choice

There is one behavior that matters a lot if you're building agents: K3 was trained with preserved thinking history.

When the assistant has already made a tool call or completed a reasoning turn, send the complete previous assistant message back on the next turn, including:

reasoning_content
content
tool_calls

Saving only the visible content throws away information the model expects to see in a multi-turn agent session.

Tool-call output should also be schema-validated before execution. vLLM notes that K3 can occasionally emit a tool-call shape that its parser does not accept cleanly. If the generated call is malformed, reject or retry it rather than sending malformed arguments into a real tool.

K3 can take images too

The native checkpoint is multimodal, so the same server can accept text and image input through the OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
    timeout=3600,
)

response = client.chat.completions.create(
    model="kimi-k3",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/image.png"
                    }
                },
                {
                    "type": "text",
                    "text": "Explain what is happening in this image."
                }
            ]
        }
    ],
    max_tokens=2048,
)

print(response.choices[0].message.content)

That makes one K3 deployment useful for coding agents that also need to inspect screenshots, diagrams, UI states, or other visual input.

When 8 GPUs are no longer enough

For larger shared services, NVIDIA Dynamo publishes K3 deployments on GB200 and GB300 NVL72 systems:

16× GB200    aggregated TP16
32× GB200    16 prefill + 16 decode

16× GB300    two TP8 replicas
24× GB300    8 prefill + 16 decode

Those are not just bigger versions of the vllm serve command above. Dynamo adds routing, distributed serving, and KV transfer between workers.

If you're bringing up one K3 endpoint, start with 8× B300/GB300 or 8× MI355X. Dynamo starts making sense when the problem changes from "run the model" to "serve substantial shared traffic."

Which K3 setup should you choose?

If you're selecting GPUs for K3 today, the simplest documented choices are:

NVIDIA: use 8× B300 or 8× GB300 with the native Moonshot checkpoint.

B200: the published route is 16× B200, using TP8 × PP2.

AMD: use 8× MI355X or MI350X with the native checkpoint.

The alternative NVFP4 and Quark checkpoints are worth testing when you have a specific reason to optimize around those formats. DSpark is worth benchmarking once the baseline server is stable.

H200, H100, A100, and 8× B200 should not be treated as equivalent alternatives just because their aggregate memory looks plausible. The current upstream deployment evidence is not at the same level.

For Kimi K3, a useful GPU setup is one that can reproduce the model load, context length, reasoning parser, tool calls, multimodal input, concurrency, and sustained generation on a pinned runtime. That's the standard that matters once you're paying for the GPUs.

Sources

Moonshot AI

• Kimi K3 official model card
  https://huggingface.co/moonshotai/Kimi-K3

• Moonshot AI
  https://www.moonshot.ai/

vLLM

• Official Kimi K3 deployment recipe
  https://recipes.vllm.ai/moonshotai/Kimi-K3

• Kimi K3 launch guide and benchmarks
  https://vllm.ai/blog/2026-07-27-k3

• Current H200 deployment issue
  https://github.com/vllm-project/recipes/issues/698

NVIDIA

• NVIDIA Dynamo Kimi K3 production recipes
  https://docs.nvidia.com/dynamo/dev/recipes/kimi-k3

AMD

• AMD Kimi K3 Quark MXFP4 + FP8 checkpoint
  https://huggingface.co/amd/Kimi-K3-Quark-MXFP4-AttnFP8

Alternative Quantization

• Red Hat Kimi K3 NVFP4
  https://huggingface.co/RedHatAI/Kimi-K3-NVFP4