Deployment recipes: single GPU, TP, DP, multi-node, load balancers and security

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

vllm serve has one command line, but it starts different sets of processes. The API server can run inside the CLI process or in several child processes. The EngineCore can hold the worker itself or drive one worker process per GPU. A node can also run only workers or only engine cores, with no HTTP port at all (--headless). The page The process and thread architecture of vllm serve explains the processes; this page gives the command for each common shape and what it starts.

The second half of the page is about production. It shows how a load balancer sees a server (/health), how to start and stop it, and which routes need a proxy. --api-key alone does not protect a vLLM server.

An everyday comparison. Think of a kitchen. One cook can do everything at one stove. A large dish can need several cooks at one stove (tensor parallelism). A busy restaurant opens more stoves, each with its own cooks (data parallelism), and a host at the door spreads the guests (the load balancer). The comparison stops at the door. vLLM's lock (the API key) closes only some of the entrances, so the building needs its own guard (a proxy).

The picture#

Read each shape from its command down. Blue boxes serve HTTP (or supervise), violet boxes schedule, green boxes own GPUs. The counts help you size the CPU: each box is at least one busy process.

In the code#

Step 1: how vllm serve picks the shape#

After the flags are parsed, ServeSubcommand.cmd chooses one of four launchers (entrypoints/cli/serve.py: the vllm serve subcommand and its launch modes):

        if is_multi_port:
            run_dp_supervisor(args)
        elif args.api_server_count < 1:
            run_headless(args)
        elif args.api_server_count > 1 or rust_frontend_path:
            run_multi_api_server(args)
        else:
            # Single API server (this process).
            args.api_server_count = None
            uvloop.run(run_server(args))

• run_server: one API server inside the CLI process. This is the default for any TP and PP with DP = 1.
• run_multi_api_server: more than one API server. The default count is the DP size with internal load balancing (vllm/entrypoints/cli/serve.py:113-130).
• run_headless: no API server (--headless sets the count to 0, vllm/entrypoints/cli/serve.py:69-77).
• run_dp_supervisor: one external-LB server per local DP rank, with an aggregated health port (--data-parallel-multi-port-external-lb, vllm/engine/arg_utils.py:1284-1292).

The engine side follows the parallel sizes. The worker runs inside the engine core (the uni executor) only when TP, PP and DP are all 1 (vllm/config/parallel.py:1055, vllm/config/parallel.py:1099-1100). With any larger size, each engine core starts one worker process per GPU (Executors and workers: from the engine core to the GPUs).

Step 2: one GPU, and tensor parallelism on one node#

# Shape A: one GPU (2 processes)
vllm serve Qwen/Qwen3-8B
# Shape B: one model over 2 GPUs of one node (4 processes)
CUDA_VISIBLE_DEVICES=0,1 vllm serve Qwen/Qwen3-8B --tensor-parallel-size 2

-tp and -pp default to 1 (vllm/config/parallel.py:142, vllm/config/parallel.py:140). With TP = 2, the engine core uses the multiprocessing executor and two worker processes, Worker_TP0 and Worker_TP1 (Executors and workers: from the engine core to the GPUs). The attention heads must divide by the TP size (Tensor and pipeline parallelism in practice).

Step 3: data parallelism on one node, and DP + EP for MoE models#

# Shape C: two independent engine copies behind one port (8 processes)
vllm serve Qwen/Qwen3-8B --data-parallel-size 2
# More front-end processes for the same engines
vllm serve Qwen/Qwen3-8B --data-parallel-size 2 --api-server-count 4
# A MoE model: 8 DP ranks, experts spread over all 8 GPUs
vllm serve Qwen/Qwen3-30B-A3B --data-parallel-size 8 --enable-expert-parallel

With -dp N on one node, the CLI process becomes a supervisor. It starts N API servers on one port, a DP coordinator, and N engine cores (Data parallel serving: several engine cores behind one endpoint). For a dense model, the engine cores are independent: each rank is set up like DP = 1 (vllm/v1/engine/core.py:1534-1543). For a MoE model, the ranks are DPEngineCoreProcs that step together in waves. --enable-expert-parallel (off by default, vllm/config/parallel.py:190) shards the experts over the ranks instead of over the TP group (Expert parallelism and MoE serving). The DP coordinator runs for MoE models, and for dense models with internal or hybrid load balancing (vllm/config/vllm.py:854-874).

Step 4: one model over several nodes#

A model that does not fit one node needs TP or PP across nodes. With the default multiprocessing backend, run one command per node. The vLLM docs give this example for 2 nodes with 8 GPUs each (docs/serving/parallelism_scaling.md:143-159):

# node 0 (the head)
vllm serve /path/to/model --tensor-parallel-size 8 --pipeline-parallel-size 2 \
  --nnodes 2 --node-rank 0 --master-addr <HEAD_NODE_IP>
# node 1
vllm serve /path/to/model --tensor-parallel-size 8 --pipeline-parallel-size 2 \
  --nnodes 2 --node-rank 1 --master-addr <HEAD_NODE_IP> --headless

--nnodes, --node-rank and --master-addr are parallel flags (vllm/engine/arg_utils.py:1165-1168). --nnodes must divide the total world size, and it needs the mp DP backend (vllm/engine/arg_utils.py:2379-2402). On node 1, run_headless sees that this node is not the first node of its DP group. Then it only starts a MultiprocExecutor for the local workers, with no engine core and no API server (vllm/entrypoints/cli/serve.py:218-231). The alternative is a Ray cluster with --distributed-executor-backend ray and one command on the head node (docs/serving/parallelism_scaling.md:122-130).

Step 5: data parallelism over several nodes#

To run DP ranks on several nodes with one HTTP endpoint, start the head node with the API servers and the other nodes with --headless. The vLLM docs give this example for DP = 4 over 2 nodes (docs/serving/data_parallel_deployment.md:40-49):

# node 0, address 10.99.48.128: API servers + DP ranks 0 and 1
vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 2 \
  --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345
# node 1: DP ranks 2 and 3, no API server
vllm serve $MODEL --headless --data-parallel-size 4 --data-parallel-size-local 2 \
  --data-parallel-start-rank 2 \
  --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345

The head node runs the API servers (by default one per DP rank, so 4 here), the DP coordinator, and its local engine cores (vllm/entrypoints/cli/serve.py:124-130). On node 1, run_headless starts only the local engine cores, which connect to the head address (vllm/entrypoints/cli/serve.py:235-256).

Step 6: disaggregated prefill and decode#

Disaggregated serving runs prefill and decode on separate vLLM instances and moves the KV cache with a KV connector. Each instance is one of the shapes above, with --kv-transfer-config (vllm/engine/arg_utils.py:1870-1872) and a router in front. The vLLM docs list the reasons, for example control of tail ITL (docs/features/disagg_prefill.md:8-13). The page Disaggregated prefill: prefill and decode on different instances explains the request flow, and KV connectors the connectors.

Step 7: behind a load balancer: independent servers, /health and readiness#

For a dense model, the simplest scale-out is a set of independent servers behind any HTTP load balancer. vLLM refuses the external DP flags for dense models and says why:

        if (
            self.data_parallel_size > 1
            and data_parallel_external_lb
            and not model_config.is_moe
        ):
            raise ValueError(
                "Non-MoE models do not support external data parallel mode. "
                "For external load balancing, launch independent vLLM "
                "instances without --data-parallel-* arguments."
            )

So run one plain vllm serve per GPU group, each with its own port or host, and no --data-parallel-* flag. For MoE models, external load balancing keeps one DP group: each server gets --data-parallel-size N --data-parallel-rank i and its own port (docs/serving/data_parallel_deployment.md:104-127).

The load balancer needs a health signal:

@router.get("/health", response_class=Response)
async def health(raw_request: Request) -> Response:
    """Health check."""
    client = engine_client(raw_request)
    if client is None:
        # Render-only servers have no engine; they are always healthy.
        return Response(status_code=200)
    try:
        await client.check_health()
        return Response(status_code=200)
    except EngineDeadError:
        return Response(status_code=503)

• 200 while the engine runs. A render-only server has no engine and always answers 200 (line 26-28).
• 503 after engine death: check_health() raises EngineDeadError once the engine has failed (vllm/v1/engine/async_llm.py:1129-1134). A watchdog then stops the server within about 5 s, unless VLLM_KEEP_ALIVE_ON_ENGINE_DEATH=1 (vllm/entrypoints/launchers/launcher.py:314-337). An orchestrator should restart it.
• No answer during start-up. The API server binds its socket first, then builds the engine, and starts uvicorn last (vllm/entrypoints/launchers/launcher.py:365-379, vllm/entrypoints/launchers/api_server/entry.py:179-201). Until uvicorn runs, the socket is bound but does not listen, so no HTTP request gets an answer (derived; uvicorn calls listen() when it starts, vllm/entrypoints/launchers/launcher.py:84-94). A start-up can take minutes (model download, compile, graph capture).

The readiness probe for vllm serve is GET /health; the vLLM docs use it for both the liveness and the readiness probe in Kubernetes (docs/deployment/k8s.md:244-255). GET /v1/models also answers only when the server is up, and it confirms the served name. The docs warn that a startup probe with a low failureThreshold kills the container during start-up (docs/deployment/k8s.md:436-443). The log then shows "KeyboardInterrupt: terminated". That text comes from the SIGTERM handler that run_server installs during start-up (vllm/entrypoints/launchers/api_server/entry.py:170-175).

Shutdown. --shutdown-timeout N (default 0) lets running requests finish for up to N seconds after a stop signal; 0 aborts them (vllm/engine/arg_utils.py:1930-1935, vllm/config/vllm.py:453-457). Set it a little below the grace period of your orchestrator (a practical rule).

Step 8: Docker, Kubernetes and the recipes folder#

The repository documents containers and orchestrators; this site does not repeat them:

• Docker: the image vllm/vllm-openai, run with --ipc=host so that processes can share memory (docs/deployment/docker.md:20-30).
• Kubernetes: a full example with a PVC, probes and /dev/shm (docs/deployment/k8s.md:1-12). Also nginx in front of several servers (docs/deployment/nginx.md), Helm (docs/deployment/frameworks/helm.md), and integrations such as llm-d, KServe and the production stack (docs/deployment/integrations/).
• recipes/: one YAML deployment recipe per model, at recipes/models/<org>/<repo>.yaml, rendered on recipes.vllm.ai (recipes/README.md:1-12). The schema lives in the separate vllm-project/recipes repository; manifest.yaml only declares the schema version (recipes/manifest.yaml:1-4). A recipe lists, for example, the minimum vLLM version and the extra arguments for tool calling. It also lists variants with --tensor-parallel-size and the needed GPU memory (recipes/models/ByteDance-Seed/Seed-OSS-36B-Instruct.yaml:12-48). Use it as a starting command for that model; the folder holds 186 files under models/ at this commit.

Step 9: security: what --api-key protects#

A secure deployment therefore puts a proxy in front of vLLM (a practical rule that the vLLM docs also give, docs/usage/security.md:280-286):

1. Let the proxy pass only the routes your clients need, for example /v1/chat/completions and /v1/models. Block /invocations, /ping, /scale_elastic_ep, /generative_scoring and everything else from outside.
2. Do the authentication and rate limits in the proxy. Keep --api-key as a second check.
3. Expose /health and /metrics only to your orchestrator and your monitoring.
4. Bind vLLM to a private address (--host), not to all interfaces.

Other switches that matter for security, all off by default:

Media URLs are fetched by the server. Limit them with --allowed-media-domains (vllm/config/model.py:235-237) and VLLM_MEDIA_URL_ALLOW_REDIRECTS=0 (default 1, vllm/envs.py:992-994); a single download is limited to VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB, default 256 (vllm/envs.py:987-989). VLLM_MAX_N_SEQUENCES (default 16384) caps n per request (vllm/envs.py:1140-1142). Traffic between nodes (PyTorch distributed, KV transfer, DP) is not encrypted; the docs require an isolated network for it (docs/usage/security.md:3-10).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

What can go wrong#

• "Non-MoE models do not support external data parallel mode." You used --data-parallel-rank or --data-parallel-external-lb with a dense model; start independent servers instead (Step 7).
• Pods restart during start-up. The probe gives up before the model is loaded and compiled; raise the startup probe threshold (Step 7).
• A dead engine stays in the pool. /health answers 503, but the load balancer checks only the TCP port. Use an HTTP health check on /health.
• Text generation without a key. /invocations is open on every server (Step 9).
• Multi-node hangs at start-up. Wrong network interface or blocked ports; see Troubleshooting vLLM: symptom, cause and fix.
• Two servers on one GPU fail to start. Each --gpu-memory-utilization is a share of the whole GPU, and the second server finds too little free memory (vllm/v1/worker/utils.py:761-770).

Why it matters. The shape decides the process count, the CPU need, the failure domain and which signals you can read. The security section decides who can use your GPUs. Both are deployment decisions that no tuning flag can fix later.