Processes, ZMQ and shared memory

BeginnerLearn the basicsabout 7 min read + 9 min deep dives

Code version: vLLM @ 3e44c0d · 2026-10-22

In one sentence

vllm serve runs as several processes that talk through ZMQ sockets and a shared-memory ring buffer, so CPU work does not slow the GPU loop.

The idea#

Why one engine uses several processes#

Python runs the bytecode of only one thread at a time in a process: the global interpreter lock (GIL) does this. A serving engine has a lot of CPU work. It parses HTTP and JSON, applies the chat template, tokenizes and detokenizes text, and schedules the next step. If all of this shared one process with the loop that feeds the GPU, the GPU would wait for Python.

vLLM therefore splits the work. The design document says that V1 "uses a multi-process architecture to separate concerns and maximize throughput" (docs/design/arch_overview.md:69):

The API server process (the front end) does the HTTP and text work.
The engine core process runs the scheduler loop. It does no HTTP or text work, so it can start the next step at once. With one GPU, it also holds the worker.
A worker process owns one GPU when there are two or more GPUs per engine.
For more front-end CPU, --api-server-count starts several API server processes that share the same engine cores (docs/design/arch_overview.md:75).

A crash or a slow request in one process also stays in that process. For example, a slow chat template does not stop the engine loop.

How processes talk#

Processes do not share Python objects. To send an object, one process turns it into bytes, and the other process turns the bytes back into an object. vLLM uses three channels:

Requests in: a ZMQ socket pair ROUTER → DEALER, from the front end to the engine core.
Outputs back: a ZMQ socket pair PUSH → PULL, from the engine core to the front end.
Work for the GPUs: a shared-memory ring buffer, MessageQueue, from the engine core to all workers.

Channels 1 and 2 carry msgpack bytes. Channel 3 carries Python pickle bytes. With one GPU there is no channel 3: the worker is an object inside the engine core process.

An everyday comparison#

Think of a restaurant again. The waiter puts each order slip into a slot in the kitchen wall (channel 1). The kitchen puts ready dishes on a counter for the waiter (channel 2). The head chef writes the next round on one whiteboard that all cooks can read at the same time (channel 3).

The comparison stops here: a slip is the order itself. A ZMQ message is only a copy in bytes, and the receiver must decode it into a new object.

The picture#

Figure: the three channels of vllm serve with TP = 2. ZMQ sockets join the front end and the engine core (panel 1). A shared-memory ring buffer joins the engine core and the workers (panel 2).

Open full size ↗

Read each panel from left to right. Panel 1 shows the frames of one message on its way to the engine core. Panel 2 shows that the engine core writes each step once, and all workers read the same bytes. The footnotes list what the picture leaves out.

In vLLM#

This is a basics page, so this part gives only pointers. The pages The process and thread architecture of vllm serve, The engine client: AsyncLLM in the API server and Executors and workers: from the engine core to the GPUs explain the code in full.

Front-end sockets. MPClient binds a ROUTER socket for inputs and a PULL socket for outputs (vllm/v1/engine/core_client.py:672-682). It encodes requests with MsgpackEncoder and decodes outputs with MsgpackDecoder(EngineCoreOutputs) (vllm/v1/engine/core_client.py:731-732).
Engine-core sockets. The engine core runs two IO threads next to its busy loop. The code comment gives the reason: socket IO releases the GIL, so the threads overlap with the GPU work (vllm/v1/engine/core.py:1238-1265).
Worker queue. MultiprocExecutor makes one MessageQueue for all workers (vllm/v1/executor/multiproc_executor.py:165-172). Each worker reads it in worker_busy_loop() (vllm/v1/executor/multiproc_executor.py:1041-1048).

vllm/v1/engine/core.py:1238-1265GitHub ↗

            # Background Threads and Queues for IO. These enable us to
            # overlap ZMQ socket IO with GPU since they release the GIL,
            # and to overlap some serialization/deserialization with the
            # model forward pass.
            # Threads handle Socket <-> Queues and core_busy_loop uses Queue.
            ready_event = threading.Event()
            input_thread = threading.Thread(
                target=self.process_input_sockets,
                args=(
                    addresses.inputs,
                    addresses.coordinator_input,
                    identity,
                    ready_event,
                ),
                daemon=True,
            )
            input_thread.start()

            self.output_thread = threading.Thread(
                target=self.process_output_sockets,
                args=(
                    addresses.outputs,
                    addresses.coordinator_output,
                    self.engine_index,
                ),
                daemon=True,
            )
            self.output_thread.start()

The engine core starts an input thread and an output thread. The busy loop on the main thread only reads and writes Python queues.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

Deep dive

Start methods: fork, spawn and what vLLM selects

Python's multiprocessing has three start methods. The vLLM design document compares them (docs/design/multiprocessing.md:22-46):

fork copies the parent process. It is fast, but it does not work well with libraries that already started threads or CUDA.
spawn starts a new interpreter that imports its modules again. It works with CUDA. But code that uses vLLM as a library needs an if __name__ == "__main__": guard.
forkserver has the same guard problem as spawn.

vLLM reads the method from VLLM_WORKER_MULTIPROC_METHOD, with the default fork (vllm/envs.py:938-940). The real choice follows these steps:

The CLI sets spawn. When you run the vllm command and the variable is not set, cli_env_setup() sets it to spawn (vllm/entrypoints/common/startup.py:147-149). The comment explains that vLLM sets it only here, because spawn can break library code without a __main__ guard (vllm/entrypoints/common/startup.py:137-140). vllm snapshot skips this setup (vllm/entrypoints/cli/main.py:67-68).
vLLM forces spawn when fork is not safe. _maybe_force_spawn() switches to spawn inside a Ray actor, with --numa-bind, when CUDA or XPU is already initialized, or on WSL (vllm/utils/system_utils.py:123-162).
Each process manager asks for the context. get_mp_context() runs the check above and returns multiprocessing.get_context(method) (vllm/utils/system_utils.py:185-200). The engine core processes (vllm/v1/engine/utils.py:256) and the worker processes (vllm/v1/executor/multiproc_executor.py:723) use it.
Extra API server processes always use spawn. APIServerProcessManager takes multiprocessing.get_context("spawn") directly (vllm/v1/utils.py:204, vllm/v1/utils.py:248-252).

So with the default vllm serve, every child process starts with spawn. Each child imports vLLM again, so each child pays the import time of vLLM once more.

Deep dive

ZMQ socket types and addresses

ZMQ is a message library. A socket sends and receives whole messages, and a message can have several frames. The ZMQ socket type decides the pattern. vLLM uses these types:

Type	Pattern	Where vLLM uses it
ROUTER (front end) → DEALER (engine core)	The ROUTER sends each message to one peer, chosen by an identity frame. The DEALER sets its identity.	Request input: the front end binds ROUTER (`vllm/v1/engine/core_client.py:672-678`). Each engine core connects a DEALER with its identity (`vllm/v1/engine/core.py:1936-1941`).
PUSH (engine core) → PULL (front end)	One-way. PUSH sends, PULL receives.	Outputs: the front end binds PULL (`vllm/v1/engine/core_client.py:680-682`); the engine core connects PUSH (`vllm/v1/engine/core.py:2047-2050`).
XPUB → XSUB	Publish to all subscribers. XPUB also sees the subscribe messages.	The DP coordinator publishes load statistics and wave state (`vllm/v1/engine/coordinator.py:214-232`). Only with DP > 1.
XPUB (in `MessageQueue`)	Publish to all readers	Messages that do not fit into a ring slot, and readers on other nodes (`vllm/distributed/device_communicators/shm_broadcast.py:522-529`, `vllm/distributed/device_communicators/shm_broadcast.py:550-556`)

Who binds. make_zmq_socket() binds by default for all types except PUSH, SUB and XSUB, which connect (vllm/utils/network_utils.py:366-367). It also sets the high-water marks of ROUTER, DEALER, PUSH and PULL to 0, which means no message limit (vllm/utils/network_utils.py:371-377).

Addresses. Sometimes all engine cores run on the same node as the front end. Then the addresses are ipc:// paths: Unix sockets in VLLM_RPC_BASE_PATH (default: the temp folder) with a random UUID (vllm/v1/engine/utils.py:1148-1151, vllm/utils/network_utils.py:175-177, vllm/envs.py:731-733). Otherwise they are tcp://host:0, and the system picks the port (vllm/utils/network_utils.py:168-172).

Identities. The identity of an engine core is its DP rank as 2 little-endian bytes (vllm/v1/engine/core_client.py:749-751). So rank 0 is b"\x00\x00" and rank 3 is b"\x03\x00". The front end puts the identity first in each message (vllm/v1/engine/core_client.py:1287-1294). ROUTER uses that frame to pick the peer and removes it, so the engine core receives (type, data...) (vllm/v1/engine/core.py:1979-1986).

Request types. The type frame is one byte: ADD = 0x00, ABORT = 0x01, START_DP_WAVE = 0x02, UTILITY = 0x03 (vllm/v1/engine/__init__.py:300-308).

Deep dive

msgpack with msgspec, and tensors out of band

The messages between the front end and the engine core are msgspec.Struct classes. EngineCoreRequest uses array_like=True: msgpack writes it as a list of values in field order, without field names. The class also sets omit_defaults=True (vllm/v1/engine/__init__.py:113-128). For an array-like struct, msgspec still writes every field; the decoder accepts a shorter list and fills missing trailing fields with their defaults (general msgspec behaviour). EngineCoreOutputs uses the same options (vllm/v1/engine/__init__.py:266-271).

MsgpackEncoder.encode() returns a list of buffers, not one bytes object (vllm/v1/serial_utils.py:165-177):

Buffer 0 is the msgpack bytes of the object.
A CPU tensor or a NumPy array smaller than VLLM_MSGPACK_ZERO_COPY_THRESHOLD (default 256 bytes) goes inline into buffer 0 (vllm/envs.py:1704-1706, vllm/v1/serial_utils.py:242-245, vllm/v1/serial_utils.py:261-263).
A larger CPU tensor stays in its own memory. The encoder appends a view of it as an extra buffer, and writes only its index into buffer 0 (vllm/v1/serial_utils.py:266-270).
ZMQ sends all buffers as frames with copy=False, so large tensors are not copied into the message (vllm/v1/engine/core_client.py:1291-1294).

Worked example (illustrative shape). A multimodal request carries one image tensor of shape [3, 336, 336] in float16: 3 × 336 × 336 × 2 = 677,376 bytes. This uses the default mm_tensor_ipc = "direct_rpc" (vllm/config/multimodal.py:440-444); with "torch_shm", the tensor goes through a separate shared-memory queue instead (vllm/v1/engine/core_client.py:725-728). This is more than 256 bytes, so the image goes as buffer 1, a separate frame after the msgpack frame. Buffer 0 holds only ("float16", (3, 336, 336), 1): the dtype, the shape and the buffer index (vllm/v1/serial_utils.py:271-272). A 32-token prompt is a list of 32 ints inside buffer 0.

Deep dive

The shared-memory MessageQueue between the engine core and the workers

With TP = 2 or more on one node, the engine core uses MultiprocExecutor. Each step, it sends the same call to every worker. A ZMQ message per worker would copy the data once per worker. MessageQueue writes it once into shared memory, and all workers read it.

The ring buffer. ShmRingBuffer has one writer and n_reader readers (vllm/distributed/device_communicators/shm_broadcast.py:265-279). Its memory has two parts (vllm/distributed/device_communicators/shm_broadcast.py:281-290):

data: max_chunks × max_chunk_bytes;
metadata: max_chunks × (1 + n_reader) bytes. Byte 0 of a chunk is the "written" flag. Then there is one "read" flag per reader.

The writer may write a chunk when no one wrote it yet, or when all readers have read it. A reader may read a chunk that is written and that it did not read yet (vllm/distributed/device_communicators/shm_broadcast.py:292-310). So there is no lock.

Sizes. MessageQueue defaults to 10 chunks (vllm/distributed/device_communicators/shm_broadcast.py:492). MultiprocExecutor sets the chunk size from VLLM_MQ_MAX_CHUNK_BYTES_MB, default 16 (vllm/v1/executor/multiproc_executor.py:152, vllm/envs.py:1856-1858). With TP = 2, the buffer is 10 × 16 MiB = 160 MiB of data plus 10 × 3 = 30 flag bytes.

One write, step by step (enqueue(), vllm/distributed/device_communicators/shm_broadcast.py:900-966):

Pickle the object. The bytes of CPU tensors become pickle buffers. A buffer of 1 MiB or more goes out of band, as a separate buffer (lines 906-914, 915-922).
If all buffers fit into one chunk, write 0 (no overflow), the buffer count and each buffer with its length into the chunk (lines 947-961).
Otherwise write only the overflow flag 1, and send the buffers through the XPUB socket (lines 943-946).
Wake the readers (line 963).

The executor puts (method, args, kwargs, output_rank) into the queue (vllm/v1/executor/multiproc_executor.py:428). Each worker calls getattr(self.worker, method) with the arguments (vllm/v1/executor/multiproc_executor.py:1055-1064). Each worker has its own MessageQueue(1, 1) for its reply (vllm/v1/executor/multiproc_executor.py:625), and the executor reads only the replies that it needs (vllm/v1/executor/multiproc_executor.py:430-449).

The model tensors between GPUs (for example the all-reduce of TP) do not use this queue. They use NCCL or a custom all-reduce. The page Executors and workers: from the engine core to the GPUs explains the executor.

Deep dive

Worked example: how many processes?

The design document gives a formula: API servers + engine cores + GPU workers, plus one DP coordinator when DP > 1 (docs/design/arch_overview.md:105-113).

Command	API servers	Engine cores	Worker processes	Coordinator	Total
`vllm serve M` (1 GPU)	1 (the CLI process)	1	0: the worker is inside the engine core	0	2
`vllm serve M -tp 4`	1	1	4	0	6
`vllm serve M -tp 2 -dp 4`	4 (default = DP size), plus the CLI process that watches them	4	8	1	18

The first row follows the code: with world size 1, the executor backend is uni (vllm/config/parallel.py:1099-1100), and UniProcExecutor makes the worker inside the engine core process (vllm/v1/executor/uniproc_executor.py:51-69). With one API server, the server runs in the CLI process (vllm/entrypoints/cli/serve.py:154-156). The API server count defaults to the DP size (vllm/entrypoints/cli/serve.py:124-130). With more than one API server, the CLI process does not serve HTTP. It starts the API server processes and waits for them, so it is one more process (vllm/entrypoints/cli/serve.py:366-390).

What can go wrong#

A script without a __main__ guard. With spawn, the child imports your main module again. Code at the top level runs again, and Python raises an error about the "bootstrapping phase" (docs/design/multiprocessing.md:126-146). Put your code under if __name__ == "__main__":.
CUDA before fork. When your process already initialized CUDA, vLLM switches to spawn with a warning (vllm/utils/system_utils.py:145-162).
A full or small /tmp or /dev/shm. The ipc sockets live in VLLM_RPC_BASE_PATH (vllm/envs.py:731-733). The ring buffer checks for free shared memory before it makes the buffer (vllm/distributed/device_communicators/shm_broadcast.py:332-335). Docker gives a small /dev/shm by default, so set a larger one.
The engine core does not start in time. The front end waits for a ready message from each engine core. It raises TimeoutError after VLLM_ENGINE_READY_TIMEOUT_S seconds (vllm/v1/engine/core_client.py:757-761).
A worker fails. When a worker replies with an error status, the executor raises RuntimeError("Worker failed with error ..."). When the call has a time limit and the reply does not come in time, it raises TimeoutError (vllm/v1/executor/multiproc_executor.py:440-448).
Insecure serialization. VLLM_ALLOW_INSECURE_SERIALIZATION=1 allows pickle in ZMQ messages (vllm/v1/serial_utils.py:220-234). Use it only on a trusted network.

Why it matters. CPU limits, shared memory, ports and start-up errors all depend on this process layout. When you know the channels, you can size the machine and read the logs of each process.

Check yourself#

1
Which ZMQ socket types carry requests from the front end to the engine core, and outputs back?
easy
Show answer
Requests: ROUTER in the front end (vllm/v1/engine/core_client.py:672-678) to DEALER in the engine core (vllm/v1/engine/core.py:1936-1941). Outputs: PUSH in the engine core (vllm/v1/engine/core.py:2047-2050) to PULL in the front end (vllm/v1/engine/core_client.py:680-682).
2
You start vllm serve from the command line and do not set VLLM_WORKER_MULTIPROC_METHOD. Which start method do the workers use?
easy
Show answer
spawn. The CLI sets the variable to spawn when it is not set (vllm/entrypoints/common/startup.py:147-149), and get_mp_context() reads it (vllm/utils/system_utils.py:185-200).
3
A request carries a CPU tensor of 100 bytes and one of 10,000 bytes. How does the encoder send each?
medium
Show answer
The 100-byte tensor is below the 256-byte threshold, so it goes inline in the msgpack buffer. The 10,000-byte tensor goes as an extra frame without a copy, and the msgpack buffer holds only its dtype, shape and frame index (vllm/v1/serial_utils.py:261-272, vllm/envs.py:1704-1706).
4
TP = 4 runs on one node. How large is the data part of the worker ring buffer? How many flag bytes does it have? What happens with a 20 MiB message?
hard
Show answer
Data: 10 chunks × 16 MiB = 160 MiB (vllm/distributed/device_communicators/shm_broadcast.py:492, vllm/v1/executor/multiproc_executor.py:152). Flags: 10 × (1 + 4) = 50 bytes (vllm/distributed/device_communicators/shm_broadcast.py:322-330). A 20 MiB message does not fit into one chunk. The writer sets the overflow flag in the chunk and sends the buffers through the XPUB socket (vllm/distributed/device_communicators/shm_broadcast.py:943-946).
5
The design document predicts 3 processes for vllm serve on one GPU. How many does the code start, and why?
hard
Show answer
2. With world size 1, the executor backend is uni (vllm/config/parallel.py:1099-1100), so the worker is an object in the engine core process (vllm/v1/executor/uniproc_executor.py:51-69). The other process is the API server in the CLI process (vllm/entrypoints/cli/serve.py:154-156). The document counts one worker process per GPU (docs/design/arch_overview.md:89-91).

Processes, ZMQ and shared memory

The idea#

Why one engine uses several processes#

How processes talk#

An everyday comparison#

The picture#

In vLLM#

Deep dive#

Start methods: fork, spawn and what vLLM selects

ZMQ socket types and addresses

msgpack with msgspec, and tensors out of band

The shared-memory MessageQueue between the engine core and the workers

Worked example: how many processes?

What can go wrong#

Check yourself#

Related pages