The process and thread architecture of vllm serve

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

An inference server does two kinds of work. CPU work: parse HTTP, apply the chat template, tokenize, detokenize and write the stream. GPU work: plan a batch, run the model and sample tokens. The GPU must never wait for the CPU work of the server. So vLLM puts the two kinds of work into different processes. The design doc says the multi-process architecture "separate[s] concerns and maximize[s] throughput" (docs/design/arch_overview.md:69).

vllm serve has three kinds of process:

• The API server process (the front end). It runs FastAPI on uvicorn, the renderer and AsyncLLM. AsyncLLM is the EngineClient of the server.
• The engine core process. It runs the scheduler, the KV cache manager and the executor in a busy loop. There is one engine core process for each data parallel rank.
• The worker processes. Each one owns one GPU and holds a model runner. With one GPU and no parallelism, there is no worker process: the worker is an object inside the engine core process.

The processes talk through three channels. ZMQ sockets with msgpack bytes connect the API servers and the engine cores. A shared-memory queue connects an engine core with its workers. NCCL connects the GPUs of one model copy. The page Processes, ZMQ and shared memory explains processes, sockets and serialization in general.

An everyday comparison. Think of an airport. The check-in desks (API servers) talk to the passengers and print the tickets. The control tower (engine core) decides which plane takes off next, but it does not fly a plane. The pilots (workers) fly the planes (GPUs). A large airport adds more desks, more towers and more pilots. The comparison stops here: in vLLM, each tower has its own fixed group of pilots, and every desk can send passengers to every tower.

The picture#

Read panel 1 from left to right. Blue is the API server process, violet is the engine core process and green is the GPU. Amber arrows are ZMQ messages. Panel 2 shows that the front end has one event loop, and the engine core has two I/O threads beside the busy loop. The tour Tour: Life of a chat request follows one request through this picture.

In the code#

Step 1: the API server process and AsyncLLM#

With one API server, vllm serve runs the server in the CLI process itself. With more than one, the CLI process starts the API servers as children:

        if is_multi_port:
            run_dp_supervisor(args)
        elif args.api_server_count < 1:
            run_headless(args)
        elif args.api_server_count > 1 or rust_frontend_path:
            run_multi_api_server(args)
        else:
            # Single API server (this process).
            args.api_server_count = None
            uvloop.run(run_server(args))

Each API server builds its engine client with AsyncLLM.from_vllm_config (vllm/entrypoints/launchers/api_server/entry.py:93-103). The constructor makes all front-end parts of the engine:

        self.renderer = renderer = renderer_from_config(self.vllm_config)

        # Convert EngineInput --> EngineCoreRequest.
        self.input_processor = InputProcessor(self.vllm_config, renderer)

        self.admission_stats = (
            SharedAdmissionStats(client_addresses, client_count, client_index)
            if client_addresses is not None
            and "mp_admission_counters" in client_addresses
            else None
        )

        # Converts EngineCoreOutputs --> RequestOutput.
        self.output_processor = OutputProcessor(
            renderer.tokenizer,
            log_stats=self.log_stats,
            stream_interval=self.vllm_config.scheduler_config.stream_interval,
            tracing_enabled=tracing_endpoint is not None,
            admission_stats=self.admission_stats,
        )

        # EngineCore (starts the engine in background process).
        # Hand the renderer to the client so it can start the frontend MM
        # warmup only after engine-core fork (the why is in
        # BaseRenderer.start_mm_warmup_in_background). The warmup is joined
        # by reset_mm_cache / warmup / shutdown.
        self.engine_core = EngineCoreClient.make_async_mp_client(
            vllm_config=vllm_config,
            executor_class=executor_class,
            log_stats=self.log_stats,
            client_addresses=client_addresses,
            client_count=client_count,
            client_index=client_index,
            renderer=renderer,
        )

The serving classes, for example the chat completion handler, take the renderer and the input processor from the engine client (vllm/entrypoints/generate/base/serving.py:150-151). So there is one renderer per API server process. The page The engine client: AsyncLLM in the API server explains AsyncLLM in full.

Step 2: the engine core process starts#

With one API server, the engine core client binds its sockets first. Then it starts the engine core processes and waits for them. (With several API servers, the CLI process binds the sockets and starts the engines; see Step 6.)

            else:
                # Engines are managed by this client.
                bind_addresses = get_engine_zmq_bind_addresses(vllm_config)
                self.input_socket = self.resources.input_socket = make_zmq_socket(
                    self.ctx,
                    bind_addresses.inputs[0],
                    zmq.ROUTER,
                    bind=True,
                    router_handover=enable_input_socket_handover,
                )
                self.resources.output_socket = make_zmq_socket(
                    self.ctx, bind_addresses.outputs[0], zmq.PULL
                )

                # EngineZmqAddresses contains only endpoints resolved by the
                # sockets that own the bind operation.
                addresses = EngineZmqAddresses(
                    inputs=[self.input_socket.getsockopt(zmq.LAST_ENDPOINT).decode()],
                    outputs=[
                        self.resources.output_socket.getsockopt(
                            zmq.LAST_ENDPOINT
                        ).decode()
                    ],
                )

                with launch_core_engines(
                    vllm_config, executor_class, log_stats, addresses
                ) as engine_launch:

launch_core_engines makes one process per local data parallel rank. Its target is EngineCoreProc.run_engine_core, and its name is EngineCore or EngineCore_DP<rank> (vllm/v1/engine/utils.py:281-288). The client then waits for one ready message from each engine core. The wait has a timeout of VLLM_ENGINE_READY_TIMEOUT_S seconds, 600 by default (vllm/v1/engine/core_client.py:753-771, vllm/envs.py:812-814).

The ready message is an EngineCoreReadyResponse (vllm/v1/engine/__init__.py:73-110). It carries values that the engine core computes at start-up, for example the number of KV cache blocks. The client copies them into its own config (vllm/v1/engine/core_client.py:847-873).

Step 3: inside the engine core process#

EngineCore.__init__ makes its parts in this order: executor (with workers and model), KV caches, structured output manager, scheduler:

        # Setup Model.
        self.model_executor = executor_class(vllm_config)
        self._pooler_config_logged = False
        if executor_fail_callback is not None:
            self.model_executor.register_failure_callback(executor_fail_callback)

        self.available_gpu_memory_for_kv_cache = -1

        if envs.VLLM_ELASTIC_EP_SCALE_UP_LAUNCH:
            self._eep_scale_up_before_kv_init()

        measure_encoder_output_widths(vllm_config, self.collective_rpc)

        # Setup KV Caches and update CacheConfig after profiling.
        kv_cache_config = self._initialize_kv_caches(vllm_config)
        self.structured_output_manager = StructuredOutputManager(vllm_config)

        # Setup scheduler.
        Scheduler = vllm_config.scheduler_config.get_scheduler_cls()

        if len(kv_cache_config.kv_cache_groups) == 0:  # noqa: SIM102
            # Encoder models without KV cache don't support
            # chunked prefill. But do SSM models?
            if vllm_config.scheduler_config.enable_chunked_prefill:
                logger.warning("Disabling chunked prefill for model without KVCache")
                vllm_config.scheduler_config.enable_chunked_prefill = False

        scheduler_block_size, hash_block_size = resolve_kv_cache_block_sizes(
            kv_cache_config, vllm_config
        )

        self.scheduler: SchedulerInterface = Scheduler(
            vllm_config=vllm_config,
            kv_cache_config=kv_cache_config,
            structured_output_manager=self.structured_output_manager,
            include_finished_set=include_finished_set,
            log_stats=self.log_stats,
            block_size=scheduler_block_size,
            hash_block_size=hash_block_size,
        )

The scheduler owns the KV cache manager (vllm/v1/core/sched/scheduler.py:329). EngineCoreProc adds the sockets and two I/O threads. The comment explains why: the threads release the GIL during socket I/O, so I/O and (de)serialization overlap with the GPU work:

            # Background Threads and Queues for IO. These enable us to
            # overlap ZMQ socket IO with GPU since they release the GIL,
            # and to overlap some serialization/deserialization with the
            # model forward pass.
            # Threads handle Socket <-> Queues and core_busy_loop uses Queue.
            ready_event = threading.Event()
            input_thread = threading.Thread(
                target=self.process_input_sockets,
                args=(
                    addresses.inputs,
                    addresses.coordinator_input,
                    identity,
                    ready_event,
                ),
                daemon=True,
            )
            input_thread.start()

            self.output_thread = threading.Thread(
                target=self.process_output_sockets,
                args=(
                    addresses.outputs,
                    addresses.coordinator_output,
                    self.engine_index,
                ),
                daemon=True,
            )
            self.output_thread.start()

The main thread runs the busy loop: it takes messages from input_queue, runs one engine step and puts the outputs into output_queue (vllm/v1/engine/core.py:1609-1620). The page The engine core loop: threads, queues and the busy loop explains the loop.

Step 4: workers, in the same process or in their own#

The executor backend decides where the workers run. The config chooses it in ParallelConfig.__post_init__:

        if self.distributed_executor_backend is None and self.world_size_across_dp > 1:
            # We use multiprocessing by default if world_size fits on the
            # current node and we aren't in a ray placement group.

            from vllm.v1.executor import ray_utils

            backend: DistributedExecutorBackend = "mp"

            self.distributed_executor_backend = backend
            logger.debug("Defaulting to use %s for distributed inference", backend)

        if self.distributed_executor_backend is None and self.world_size == 1:
            self.distributed_executor_backend = "uni"

The first test uses world_size_across_dp, which is world_size × data_parallel_size (vllm/config/parallel.py:627-629). The world_size is PP × TP × PCP (vllm/config/parallel.py:965-969). So the "uni" backend applies only when TP, PP and DP are all 1 (and PCP too). Then UniProcExecutor builds the worker in its own process, the engine core process (vllm/v1/executor/uniproc_executor.py:51-75).

In all other single-node cases on CUDA, the default backend is "mp". Ray replaces it when Ray is active with a placement group, or when the data parallel backend is Ray (vllm/config/parallel.py:1061-1096). MultiprocExecutor starts one process per local rank, named VllmWorker-<rank> (vllm/v1/executor/multiproc_executor.py:191-207, vllm/v1/executor/multiproc_executor.py:745-750). The page Executors and workers: from the engine core to the GPUs explains both executors.

Step 5: the three channels#

ZMQ between API servers and engine cores. The client binds a ROUTER socket for requests and a PULL socket for outputs (Step 2). The engine core connects a DEALER socket with its identity, one for each API server (vllm/v1/engine/core.py:1935-1943). It also connects one PUSH socket for each API server (vllm/v1/engine/core.py:2046-2052). The identity is the engine index as 2 little-endian bytes (vllm/v1/engine/core.py:1178-1179). The addresses are ipc:// paths when the API server sends only to engines on its own node, and TCP addresses otherwise (vllm/v1/engine/utils.py:1137-1151). The client encodes requests with MsgpackEncoder and decodes EngineCoreOutputs with MsgpackDecoder (vllm/v1/engine/core_client.py:731-732).

Shared memory between an engine core and its workers. MultiprocExecutor makes one MessageQueue with a reader for each worker (vllm/v1/executor/multiproc_executor.py:165-171). Every call, for example execute_model, goes into this queue once and reaches all workers (vllm/v1/executor/multiproc_executor.py:428). The queue pickles the object (vllm/distributed/device_communicators/shm_broadcast.py:934-940). Only one worker replies to execute_model: the first TP rank of the last PP stage (vllm/v1/executor/multiproc_executor.py:549-563).

NCCL between GPUs. Inside the forward pass, the workers of one TP group exchange activations with collectives. On CUDA, each device communicator with more than one rank makes a PyNcclCommunicator (vllm/distributed/device_communicators/cuda_communicator.py:106-111). With PP, each stage after the first receives the intermediate tensors of the stage before it. Each stage before the last sends them on (vllm/v1/worker/gpu_worker.py:1337-1342, vllm/v1/worker/gpu_worker.py:1375-1378). The page Process groups: how vLLM slices its ranks explains the groups.

Step 6: how the counts change with TP, PP, DP and API servers#

TP and PP are explained in Parallelism basics↗. In vLLM they change only the number of workers per engine core. DP changes the number of engine cores. --api-server-count changes the number of API servers. Its default depends on the load balancing mode:

        # Default api_server_count if not explicitly set.
        # - Multi-port: 1 (supervisor spawns one server per local DP rank)
        # - Rust frontend: 1 (not applicable as it's multithreaded)
        # - External LB: 1 (external LB handles distribution)
        # - Hybrid LB: Use local DP size (internal LB for local ranks only)
        # - Internal LB: Use full DP size
        if args.api_server_count is None:
            if is_multi_port or is_external_lb or rust_frontend_path:
                args.api_server_count = 1
            elif is_hybrid_lb:
                args.api_server_count = args.data_parallel_size_local or 1
                if args.api_server_count > 1:
                    logger.info(
                        "Defaulting api_server_count to data_parallel_size_local "
                        "(%d) for hybrid LB mode.",
                        args.api_server_count,
                    )
            else:
                args.api_server_count = args.data_parallel_size
                if args.api_server_count > 1:
                    logger.info(
                        "Defaulting api_server_count to data_parallel_size (%d).",
                        args.api_server_count,
                    )

So with internal load balancing, the default is one API server per DP rank. Each API server gets a client_index and the client_count (vllm/v1/utils.py:230-238). With DP > 1 and internal load balancing, the client is a DPLBAsyncMPClient that can send each request to any engine core (vllm/v1/engine/core_client.py:158-169). So the API servers and the engine cores form a many-to-many network.

The model is dense, so each DP rank is an independent copy of the model. The DP coordinator exists because the API servers balance the load with its counts. The deep dive below gives the rules for each process type.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

What can go wrong#

• The engine core does not become ready. A large model can load for a long time. After VLLM_ENGINE_READY_TIMEOUT_S (600 s by default) the client raises TimeoutError with a message that names the variable (vllm/v1/engine/core_client.py:756-768).
• A worker process dies. The worker monitor of the executor logs "Worker proc ... died unexpectedly" and calls the failure callback (vllm/v1/executor/multiproc_executor.py:314-326). The callback puts EXECUTOR_FAILED into the input queue of the engine core (vllm/v1/engine/core.py:1174-1176). The busy loop raises RuntimeError("Executor failed.") (vllm/v1/engine/core.py:1766-1767).
• The engine core dies. On a fatal error it sends ENGINE_CORE_DEAD to every API server (vllm/v1/engine/core.py:1570-1576, vllm/v1/engine/core.py:2066-2069). The client then raises EngineDeadError (vllm/v1/engine/core_client.py:581-584). The API server shuts down, unless VLLM_KEEP_ALIVE_ON_ENGINE_DEATH is set (vllm/entrypoints/launchers/launcher.py:325-337).
• Not enough GPUs. When the world size is larger than the number of GPUs on the node, the config raises ValueError (vllm/config/parallel.py:1067-1078).
• Flags that do not combine. --api-server-count > 0 with --headless raises ValueError (vllm/entrypoints/cli/serve.py:69-75). Runtime LoRA updates with more than one API server raise ValueError (vllm/entrypoints/cli/serve.py:315-318).

Why it matters. To size a server, you must know how many processes and CPU threads it needs. To debug it, you must know which process logs what. A tokenizer error appears in an API server, a scheduler error in an engine core, and a CUDA error in a worker. To scale it, you must know which flag adds which process.