Deploying DeepSeek V4 Flash 0731 with vLLM: Verified GPU Pods, Quants, and Serving Recipes

DeepSeek V4 Flash 0731 is the current official Flash release:

deepseek-ai/DeepSeek-V4-Flash-0731

It keeps the same underlying architecture as the earlier V4 Flash preview, but DeepSeek re-trained the model for much stronger agentic performance. The published results are a big jump: 82.7 on Terminal Bench 2.1 versus 61.8 for the preview, and 54.4 on DeepSWE versus 7.3.

The model is MIT-licensed and ungated.

Under the hood, V4 Flash is a 284B-parameter Mixture-of-Experts model with roughly 13B parameters active per token. It supports a native context window of 1,048,576 tokens and uses a hybrid attention design combining Compressed Sparse Attention (CSA) with Heavily Compressed Attention (HCA).

The checkpoint is already aggressively compressed. Routed experts are stored in MXFP4 with FP8 scales, while linear, attention, and router layers use FP8. At serving time, the KV cache can also be stored in FP8.

DeepSeek reports that V4 needs roughly 27% of V3.2's inference FLOPs per token and about 10% of its KV-cache footprint at 1M context.

If Hugging Face shows roughly 304–305B parameters for the repository, that does not contradict the 284B model size. The fused 0731 repository also contains the DSpark speculative-decoding module.

The checkpoint is around 167 GB on disk, compared with roughly 160 GB for the preview. Do not use those figures as direct VRAM requirements. vLLM's MI325X validation, for example, measured about 148.66 GiB of resident model memory for the preview checkpoint.

That difference is partly units: Hugging Face reports decimal GB, while GPU memory is normally discussed in GiB. More importantly, loading the weights is only part of the memory budget. KV cache, runtime buffers, kernels, and concurrency all need room too.

Which checkpoint should you use?

For most users, start with:

deepseek-ai/DeepSeek-V4-Flash-0731

The other variants are useful when a particular GPU or runtime path requires them:

One detail matters if you are choosing NVIDIA's NVFP4 checkpoint: it is based on the preview weights, not 0731. You get the NVIDIA NVFP4 execution path, but not the newer post-training that produced the large agentic gains above.

The community W4A16 conversion is also not proof that stock 0731 runs cleanly on A100. Its published end-to-end validation used Radeon Pro W6800X Duo hardware with CPU offload.

Pin vLLM before you start

The minimum runtime depends on the path you are using:

For the documented NVIDIA 0731 setup:

docker pull vllm/vllm-openai:v0.25.0

Once you have a working deployment, keep the version pinned. A model this dependent on specialized kernels is not a good place to discover that latest changed underneath you.

Which GPUs can you actually use?

The current DeepSeek and vLLM recipes give you these reproducible paths:

For ordinary multi-GPU serving on H200, B200, and B300, vLLM's recommended pattern is:

Data Parallel = 4
Expert Parallel = enabled

H100, A100, 2× H200, 2× B200, 1× B200, and MI300X are not included here because the current recipe does not provide the same level of end-to-end deployment evidence.

4× GB300: the cleanest 0731 reference

DeepSeek publishes this command for a single 4× GB300 node:

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

This is specifically a Blackwell recipe.

These two options:

--moe-backend deep_gemm_mega_moe
--attention-config '{"use_fp4_indexer_cache": true}'

should not be copied onto H200.

You may also see the FP4 indexer setting written upstream as:

--attention_config.use_fp4_indexer_cache True

Use the syntax supported by the vLLM build you pinned.

B200 and B300

For 4× B200 or 4× B300, a useful starting point is:

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice

To add DSpark:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

There is a small upstream disagreement here: DeepSeek's model card uses greedy, while the current vLLM recipe generator uses probabilistic.

If speculative decoding matters to your workload, benchmark both rather than treating either value as universally best.

H200 uses a different path

H200 is Hopper, so the Blackwell FP4 indexer and deep_gemm_mega_moe overrides are intentionally absent.

Start with:

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --max-model-len auto

If you have a full 8× H200 node, vLLM also documents a more advanced layout with four GPUs handling prefill and four handling decode. KV state is transferred using Mooncake or NIXL, with vllm-router in front.

Those published prefill/decode examples currently use the preview checkpoint, not 0731, so keep that distinction if you are reproducing them exactly.

RTX PRO 6000 Blackwell needs two extra setup steps

vLLM has an explicit 0731 validation for:

8× RTX PRO 6000 Blackwell
TP8
Expert Parallel

using:

vllm/vllm-openai:v0.25.0

Before starting the model, install the required FlashInfer build:

/usr/bin/python3.12 -m pip install \
  --no-deps \
  flashinfer-python==0.6.14

export FLASHINFER_DISABLE_VERSION_CHECK=1

The --no-deps is important. Installing FlashInfer normally can replace PyTorch and break vLLM's compiled extensions.

Then run:

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4

On this particular profile, do not enable MTP or DSpark. Both currently fail during warmup on the sm_120 sparse-MLA path.

AMD: one MI325X can run it, with limits

The single-GPU MI325X validation is one of the more useful upstream tests because vLLM publishes actual memory observations rather than just saying the model fits.

The validated setup is:

1× MI325X 256GB
TP1
4K context
10 GB explicit KV cache

Run:

export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --host 0.0.0.0 \
  --port 8002 \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 4096 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 256 \
  --kv-cache-memory-bytes 10000000000 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --tokenizer-mode deepseek_v4 \
  --moe-backend triton_unfused \
  --enforce-eager

vLLM measured about 148.66 GiB of model memory after loading and reports successful chat, non-thinking, and Think High tests over a 24-hour allocation.

This is a useful single-GPU option if 4K context is enough. It should not be interpreted as a one-GPU million-token deployment.

The published AMD validation uses the preview checkpoint. If you want 0731 with DSpark on ROCm, use vLLM 0.26.0 or newer.

4× MI355X

The other documented AMD path uses TP4 across four MI355X GPUs:

export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --host localhost \
  --port 8001 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 4 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --compilation-config '{"mode":3,"cudagraph_mode":"FULL_DECODE_ONLY"}'

The published validation also includes 8-shot GSM8K results:

Flexible-extract exact match: 0.9439
Strict-match exact match:     0.9431

TP8 is available when lower latency matters more than reproducing the published TP4 setup.

DGX Spark is a special case

A single DGX Spark GB10 has 128 GB of unified memory, so the documented path uses two machines at TP2, one GB10 per node.

It also uses a specialized image:

eugr/spark-vllm-b12x:latest

rather than the standard vLLM container, plus ConnectX/RoCE network configuration between the nodes.

If you are using DGX Spark, follow that recipe as its own deployment path. It is not equivalent to a normal two-GPU Blackwell pod.

vLLM also carries a single-GPU DGX Station GB300 profile. The currently published command uses the preview checkpoint with MTP rather than the 0731 DSpark release.

Don't size every server for 1M context

DeepSeek V4 Flash supports:

1,048,576 tokens

but that does not mean every deployment should reserve enough memory to serve a million-token request.

The MI325X profile makes the distinction obvious: the model supports 1M context, while that particular validated server configuration supports 4K.

Several vLLM recipes therefore use:

--max-model-len auto

instead of forcing the model's architectural maximum.

V4 also has a hybrid CSA+HCA cache, so generic dense-transformer calculations such as "X GB per 100K tokens" are not reliable across every configuration.

Use the memory profile vLLM prints at startup. If you need to reserve a fixed cache budget, use:

--kv-cache-memory-bytes

Most published configurations use:

--kv-cache-dtype fp8

or, on ROCm:

--kv-cache-dtype fp8_e4m3

to reduce cache memory.

If you are using --data-parallel-size 4, remember that scheduling settings such as --max-num-seqs and --max-num-batched-tokens apply per data-parallel rank, not once for the entire four-GPU server.

Reasoning and tool calling

DeepSeek V4 Flash supports three reasoning levels:

low
high
max

DeepSeek's recommended sampling depends on what you are doing:

If you are serving Think High or Think Max, leave room for very long outputs. DeepSeek recommends allowing up to 384K output tokens for the deeper reasoning modes, and the vLLM recipe notes that Think Max needs:

--max-model-len >= 393216

to avoid truncating the reasoning budget.

For the OpenAI-compatible vLLM endpoint, start the server with:

--tokenizer-mode deepseek_v4 \
--reasoning-parser deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice

A normal request looks like:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Flash-0731",
    "messages": [
      {
        "role": "user",
        "content": "Explain optimistic and pessimistic locking."
      }
    ],
    "temperature": 1.0,
    "top_p": 1.0,
    "max_tokens": 2048
  }'

For an agentic request with Think High:

from openai import OpenAI

client = OpenAI(
    base_url="http://YOUR_SERVER_IP:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash-0731",
    messages=[
        {
            "role": "user",
            "content": "Find the race condition in this worker design."
        }
    ],
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "high"
        }
    },
)

print(response.choices[0].message)

Keep the model on persistent storage

Current Hugging Face tooling uses the hf CLI:

export HF_HOME=/workspace/huggingface
mkdir -p "$HF_HOME"

hf download deepseek-ai/DeepSeek-V4-Flash-0731

At roughly 167 GB, repeatedly downloading the checkpoint every time a pod is recreated gets expensive in both time and bandwidth. Persistent model storage is worth setting up from the beginning.

Which DeepSeek V4 Flash setup should you choose?

If you are choosing GPUs specifically for DeepSeek-V4-Flash-0731, the current upstream paths are fairly clear.

4× GB300 is the cleanest reference because DeepSeek publishes the exact 0731 command.

4× B200 or B300 gives you the equivalent Blackwell DP4 + expert-parallel route.

4× H200 is the documented Hopper setup. Do not copy the Blackwell FP4-specific flags into it.

If you have 8× H200, vLLM also has a more advanced prefill/decode design, although the published commands currently use the preview checkpoint.

8× RTX PRO 6000 Blackwell is explicitly validated for 0731, but MTP and DSpark need to stay off in the current profile.

On AMD, 1× MI325X is a real validated option if a 4K context limit is acceptable, while 4× MI355X gives you the stronger multi-GPU ROCm path.

H100, A100, 2× H200, 2× B200, 1× B200, and MI300X are intentionally not recommended here. The current upstream material does not provide equivalent deployment recipes for them.

With V4 Flash, the useful question is not simply whether the checkpoint can fit in aggregate memory. You need enough headroom for the actual runtime, a supported kernel path for the GPU architecture, enough KV cache for the context you plan to serve, and a configuration that stays up under real concurrent traffic.

Sources

DeepSeek

• DeepSeek V4 Flash 0731 model card
  https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731

• DeepSeek V4 Flash preview — architecture and precision
  https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash

• DeepSeek V4 technical report
  https://arxiv.org/abs/2606.19348

vLLM

• DeepSeek V4 Flash deployment recipe
  https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash

• Raw hardware profiles and validations
  https://raw.githubusercontent.com/vllm-project/recipes/main/models/deepseek-ai/DeepSeek-V4-Flash.yaml

• FP8 KV-cache documentation
  https://docs.vllm.ai/en/latest/features/quantization/quantized\_kvcache/

Alternative checkpoints

• NVIDIA DeepSeek V4 Flash NVFP4
  https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4

• Community 0731 NVFP4 conversion
  https://huggingface.co/MJPansa/DeepSeek-V4-Flash-0731-NVFP4

• Community 0731 INT4/W4A16 conversion
  https://huggingface.co/yiminyuan/DeepSeek-V4-Flash-0731-INT4-W4A16

• Unsloth DeepSeek V4 GGUF documentation
  https://unsloth.ai/docs/models/deepseek-v4