# Deploying Qwen3.8-2.4T-A95B with vLLM: Verified GPU Pods, Quants, and Serving Recipes

Qwen3.8-2.4T-A95B is a 2.4-trillion-parameter Mixture-of-Experts model with roughly 95B parameters active for each token. If you're planning to self-host it, the first thing to know is that this is a genuinely large distributed model: even the low-precision checkpoints are measured in terabytes.

The official open checkpoint is:

```text
Qwen/Qwen3.8-2.4T-A95B
```

The model has 512 routed experts and selects 10 of them per token alongside one shared expert. Its 92-layer backbone mixes 69 Gated DeltaNet linear-attention layers with 23 full-attention layers, with full attention appearing every fourth layer.

Native context is **262,144 tokens**, with an extended configuration available up to roughly **1.01 million tokens**.

The open checkpoint is text-only and always uses reasoning. This is different from Qwen's hosted `Qwen3.8-Max` service, which adds features such as vision input and non-thinking mode.

For GPU deployment, the main decision is not whether 2.4T parameters will somehow fit. It is **which precision format gives you a documented configuration on the hardware you actually have**.

## Start with the checkpoint that matches your GPUs

The practical options today are:

| Your GPUs | Checkpoint | Documented setup |
| --- | --- | --- |
| **8× B300** | `Inferact/Qwen3.8-2.4T-A95B-NVFP4` | TP8 |
| **8× GB300** | `Inferact/Qwen3.8-2.4T-A95B-NVFP4` | TP8 across two NVL4 trays |
| **16× B300** | `Qwen/Qwen3.8-2.4T-A95B-FP8` | TP16 |
| **16× GB300** | `Qwen/Qwen3.8-2.4T-A95B-FP8` | TP16 |
| **12× GB300** | `Qwen/Qwen3.8-2.4T-A95B-FP8` | TP4 × PP3 |
| **8× MI355X** | `Inferact/Qwen3.8-2.4T-A95B-MXFP4` | TP8 |

The full BF16 checkpoint is roughly **4.45 TiB**. The official FP8 version is around **2.27 TiB**, while the NVFP4 checkpoint used in the NVIDIA eight-GPU recipe is around **1.32 TiB**.

That is why NVFP4 is the most approachable NVIDIA deployment if your goal is simply to get Qwen3.8 running without moving immediately to a 16-GPU cluster.

H100, H200, A100, B200 and smaller GPU configurations are not included here. Current vLLM material contains sizing information for some of those GPUs, but not equivalent end-to-end serving recipes. This guide does not turn memory estimates into deployment claims.

## Before starting the model

The current vLLM recipe recommends a recent nightly build rather than an older stable release.

Create an environment with:

```bash
uv venv
source .venv/bin/activate

uv pip install -U vllm \
  --extra-index-url https://wheels.vllm.ai/nightly

uv pip install -U "transformers>=5.4.0"
```

Once you have a working build, pin it. Qwen3.8 relies on recently added model and kernel support, so continuously upgrading a production server to whatever nightly happens to be current is unnecessary risk.

## Running Qwen3.8 on 8× B300 or GB300

For NVIDIA, the smallest documented configuration uses:

```text
Inferact/Qwen3.8-2.4T-A95B-NVFP4
```

with eight GPUs.

Start with:

```bash
vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \
  --tensor-parallel-size 8 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

This configuration applies to an eight-GPU B300 system and is also documented across eight GB300 GPUs spanning two NVL4 trays.

vLLM's optimized NVFP4 path also uses:

```bash
--linear-backend flashinfer_cutedsl
```

but that flag depends on having the matching FlashInfer environment. Add it when reproducing the corresponding vLLM container/software stack rather than assuming every vLLM installation has the required backend.

### Keep MTP enabled when you benchmark

Qwen3.8 contains a built-in Multi-Token Prediction head. vLLM's published results show that using three speculative tokens can make a substantial difference.

On its low-latency tests:

```text
FP8 TP16
Without MTP: 130 output tok/s/user
MTP-3:       307 output tok/s/user

NVFP4 TP8
Without MTP: 133 output tok/s/user
MTP-3:       304 output tok/s/user
```

Those are measurements from vLLM's hardware and workload, not promised performance for every server. They do make MTP-3 worth testing from the beginning:

```bash
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

Using only one speculative token performed much less convincingly in the same testing. vLLM measured 64.8% acceptance for MTP-1, and at higher concurrency the extra speculative work could actually hurt throughput.

## Running the official FP8 checkpoint

If you want Qwen's official FP8 checkpoint:

```text
Qwen/Qwen3.8-2.4T-A95B-FP8
```

the documented low-latency configuration moves to **16 GPUs**.

On B300, that normally means two eight-GPU nodes using TP16.

The head node runs:

```bash
vllm serve Qwen/Qwen3.8-2.4T-A95B-FP8 \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_ADDR \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

The second node uses the same distributed configuration with:

```bash
--node-rank 1 \
--headless
```

Only rank 0 should expose the API server.

There is also a verified **12× GB300** FP8 configuration using:

```text
TP4 × PP3
```

Why not TP12? Qwen3.8 has 64 full-attention heads, so its tensor-parallel size needs to divide 64. TP12 does not.

vLLM verified the TP4 × PP3 layout on 12 GB300 GPUs, including model loading, CUDA graph capture and generation. It is a useful option when you have three four-GPU GB300 trays, although the ordinary TP8 and TP16 setups remain simpler.

## Running it on AMD

The documented AMD route uses:

```text
Inferact/Qwen3.8-2.4T-A95B-MXFP4
```

on **8× MI355X**.

```bash
vllm serve Inferact/Qwen3.8-2.4T-A95B-MXFP4 \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

One difference from the NVIDIA command is worth preserving: the current vLLM recipe does not recommend blindly adding FP8 KV cache to this ROCm setup.

Do not assume:

```bash
--kv-cache-dtype fp8
```

works with every ROCm/vLLM combination. Add it only after confirming it on the exact software build running on the node.

AMD also publishes:

```text
amd/Qwen3.8-2.4T-A95B-Quark-MXFP4
```

for MI350 and MI355 hardware.

In that conversion, routed experts use OCP MXFP4 while components such as attention, routers, the shared expert, LM head and MTP layer remain at higher precision.

AMD's published GSM8K reproduction reported:

```text
FP8 baseline: 97.49
MXFP4:        97.49
```

That result was reproduced with SGLang at TP8 on MI35x hardware. It is evidence for the quality of AMD's quantization, not a vLLM throughput benchmark.

## How much context should you actually allocate?

The open checkpoint natively supports:

```text
262,144 tokens
```

and can be extended to about:

```text
1,010,000 tokens
```

with vLLM:

```bash
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
```

and:

```bash
--max-model-len 1010000 \
--hf-overrides '{"max_position_embeddings":1010000}'
```

There is little reason to enable 1M context automatically.

Long context consumes memory that could otherwise be used for concurrent requests. vLLM's own NVFP4 testing illustrates the tradeoff: a configuration sized around the 262K window could hold roughly 25 concurrent requests in the available KV-cache budget, while a much shorter workload around 8K input plus 1K output allowed hundreds.

If you're serving repository-scale coding agents, very long context may be worth the cost. If most requests are 10K or 20K tokens, reserving for one million tokens mostly reduces how many users the GPUs can serve at once.

Set:

```bash
--max-model-len
```

for the workload you actually expect.

## Reasoning and tool calling

Qwen3.8 is a reasoning model. The open checkpoint does not expose a normal non-thinking mode.

It supports three reasoning-effort settings:

```text
xhigh
medium
low
```

`xhigh` is the default.

Qwen also preserves thinking across turns by default, which matters for agentic sessions where earlier reasoning and tool interactions are part of the ongoing context.

Start vLLM with:

```bash
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
```

Qwen's recommended generation settings include:

```text
temperature = 1.0
top_p = 0.95
top_k = 20
min_p = 0.0
presence_penalty = 0.0
repetition_penalty = 1.0
```

A normal OpenAI-compatible client looks like:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
    timeout=3600,
)

response = client.chat.completions.create(
    model="Inferact/Qwen3.8-2.4T-A95B-NVFP4",
    messages=[
        {
            "role": "user",
            "content": "Review this distributed queue design and identify its failure modes."
        }
    ],
    reasoning_effort="medium",
    temperature=1.0,
    top_p=0.95,
    max_tokens=16384,
)

print(response.choices[0].message)
```

For hard coding and agent tasks, leave enough output budget for the reasoning trace. An aggressive `max_tokens` limit can terminate the generation before the model reaches its final answer.

## Loading a model this large

At Qwen3.8's scale, startup time becomes part of operating the server.

vLLM's NVFP4 testing found that:

```bash
--load-format fastsafetensors \
--safetensors-load-strategy lazy
```

reduced model load time from **545 seconds to 306 seconds** on the shared storage used in that test.

That result will vary with storage performance, but persistent model storage is clearly preferable to downloading or repeatedly copying a terabyte-scale checkpoint whenever a machine restarts.

vLLM also uses a larger engine startup timeout:

```bash
export VLLM_ENGINE_READY_TIMEOUT_S=3600
```

And don't make your own readiness decision solely from whether the process is alive. Send a small request to:

```text
/v1/chat/completions
```

and verify that the model can actually generate.

## Which GPU setup should you use?

If you're renting GPUs specifically for Qwen3.8-2.4T-A95B, the current documented choices are fairly straightforward.

**8× B300 + NVFP4** is the simplest NVIDIA configuration in the current vLLM recipe.

**8× GB300 + NVFP4** gives you the equivalent Grace Blackwell path across two NVL4 trays.

Use **16× B300 or GB300 + FP8** when you specifically want the official FP8 checkpoint and have enough hardware for TP16.

If you have three GB300 trays, **12× GB300 FP8 with TP4 × PP3** is an upstream-verified alternative.

On AMD, use **8× MI355X + MXFP4**.

H100, H200, A100, B200 and smaller GPU counts are intentionally not recommended here. There may be enough aggregate memory in some of those configurations, but the current sources do not provide the same reproducible end-to-end deployment evidence.

For Qwen3.8, choosing the GPUs is only part of the deployment. The exact quant, tensor-parallel layout, KV-cache budget, vLLM build, network topology and MTP configuration all affect whether the resulting server is useful once real traffic starts hitting it.

## Sources

### Qwen

*   **Qwen3.8-2.4T-A95B official model card**  
    https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B
    
*   **Official FP8 checkpoint**  
    https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8
    

### vLLM

*   **Official Qwen3.8-2.4T-A95B deployment recipe**  
    https://recipes.vllm.ai/Qwen/Qwen3.8-2.4T-A95B
    
*   **vLLM Qwen3.8 launch notes and benchmarks**  
    https://vllm.ai/blog/2026-08-12-qwen3.8
    

### NVIDIA / Inferact

*   **Qwen3.8 NVFP4 checkpoint**  
    https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4
    
*   **Qwen3.8 MXFP4 checkpoint**  
    https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4
    

### AMD

*   **AMD Qwen3.8 Quark MXFP4 checkpoint**  
    https://huggingface.co/amd/Qwen3.8-2.4T-A95B-Quark-MXFP4