Process groups: how vLLM slices its ranks

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

Parallel inference splits one model over several GPUs, and the GPUs exchange tensors with collectives such as all-reduce (Parallelism basics↗). A collective always runs inside a process group: a fixed set of ranks that take part together. Tensor parallelism needs one group, pipeline parallelism another, and expert parallelism a third.

vLLM builds all its groups in the same way, in every worker. It numbers the ranks and puts them into a grid with one dimension per kind of parallelism. Then it reads each group off one line of the grid. This order is the rank layout. TP is the last dimension, so the ranks of one TP group are neighbours, and they normally sit on one node.

Each group becomes a GroupCoordinator. It holds two PyTorch groups for the same ranks: a gloo group for small CPU messages and an NCCL group for GPU tensors. When the group has more than one rank, it also holds a device communicator. On CUDA, this object picks the fastest all-reduce for each tensor: for example the custom all-reduce, symmetric memory or plain NCCL.

An everyday comparison. Think of the seat plan of a large exam hall. Each seat has a row and a column. "Row 3" and "column 5" are two different teams, but each seat belongs to exactly one row and one column. The hall manager does not invent the teams one by one; she reads them off the plan. The comparison stops here: a vLLM rank belongs to one group of each kind at the same time. Some groups (EP) also span several rows and columns.

The picture#

Read panel 1 as the grid: each orange lane is one DP rank, which is one engine core with its four workers. Panel 2 lists the members of each group; the code moves the group's dimension to the end of the grid and cuts rows. Panel 3 shows that only groups with more than one rank get a device communicator, and only TP groups get the special all-reduce kernels.

In the code#

Step 1: the worker sets up the groups in init_device#

Each worker builds its groups in init_device, before it takes the memory snapshot. The comment says why: the NCCL buffers must exist before vLLM measures the free memory:

            current_platform.check_if_supports_dtype(self.model_config.dtype)

            # Initialize the distributed environment BEFORE taking
            # memory snapshot
            # This ensures NCCL buffers are allocated before we measure
            # available memory
            init_worker_distributed_environment(
                self.vllm_config,
                self.rank,
                self.distributed_init_method,
                self.local_rank,
                current_platform.dist_backend,
            )

init_worker_distributed_environment first prepares the batch-invariant mode and the EPLB environment variables. Then it does four things in order. It sets the custom all-reduce switch from --disable-custom-all-reduce, creates the world group, creates the model-parallel groups, and starts the EC connector (vllm/v1/worker/gpu_worker.py:1678-1720). The rank and the init method come from the multiproc executor: the global rank of a worker is local_world_size × node_rank_within_dp + local_rank (vllm/v1/executor/multiproc_executor.py:179-192). On one node, the init method is a file:// store in the temp folder, unless AITER needs a TCP store (vllm/v1/executor/multiproc_executor.py:139-144).

Step 2: init_distributed_environment makes the world group#

The first call joins all ranks of this engine into PyTorch's default group. With DP > 1 or several nodes, it first widens the world across the data parallel ranks:

    if (
        config is not None
        and config.parallel_config.distributed_executor_backend != "external_launcher"
        and (
            config.parallel_config.nnodes > 1
            or config.parallel_config.data_parallel_size > 1
        )
        and not enable_elastic_ep
    ):
        parallel_config = config.parallel_config
        # adjust to take into account data parallelism
        # offset the rank by the data parallel rank
        rank = parallel_config.data_parallel_rank * world_size + rank
        # adjust the world size to take into account data parallelism
        world_size = parallel_config.world_size_across_dp

        # Use appropriate IP and port based on configuration
        if parallel_config.nnodes > 1:
            ip = parallel_config.master_addr
            port = parallel_config.master_port

1. Line 1869. The rank is shifted by data_parallel_rank × world_size. So DP rank 1 of a 4-rank engine starts at global rank 4.
2. Line 1871. The world grows to world_size_across_dp = TP × PP × PCP × DP (vllm/config/parallel.py:626-629).
3. Lines 1874-1876. With several nodes, the rendezvous goes to --master-addr and --master-port. Otherwise it goes to the DP master address with a fresh port (lines 1878-1920).

The world group itself is a GroupCoordinator over all ranks, with no device communicator (vllm/distributed/parallel_state.py:1507-1516, vllm/distributed/parallel_state.py:2010-2012).

Step 3: initialize_model_parallel reads the groups off the grid#

    # the layout order is: ExternalDP x DP x PP x PCP x TP
    # ExternalDP is the data parallel group that is not part of the model,
    # every dp rank can generate independently (in verl integration).
    # DP is the data parallel group that is part of the model,
    # all the ranks in the same DP group should generate simultaneously,
    # i.e. the `generate` call in the same DP group should be called together,
    # otherwise it will cause deadlock.
    # to get group_ranks for each dimension, transpose that dimension to the
    # last dimension, then reshape to 2D, then unbind the last dimension
    all_ranks = torch.arange(world_size).reshape(
        -1,
        data_parallel_size,
        pipeline_model_parallel_size,
        prefill_context_model_parallel_size,
        tensor_model_parallel_size,
    )  # noqa

    # Build the tensor model-parallel groups.
    global _TP
    assert _TP is None, "tensor model parallel group is already initialized"
    group_ranks = all_ranks.view(-1, tensor_model_parallel_size).unbind(0)
    group_ranks = [x.tolist() for x in group_ranks]
    if enable_elastic_ep:
        group_ranks = local_all_ranks.view(-1, tensor_model_parallel_size).unbind(0)
        group_ranks = [x.tolist() for x in group_ranks]
    # message queue broadcaster is only used in tensor model parallel group
    _TP = init_model_parallel_group(
        group_ranks,
        get_world_group().local_rank,
        backend,
        use_message_queue_broadcaster=True,
        group_name="tp",
    )

1. Lines 2148-2156. The comment gives the layout order: ExternalDP × DP × PP × PCP × TP. ExternalDP is for frameworks that run several independent copies of vLLM in one world.
2. Lines 2157-2163. One tensor of all ranks, shaped (-1, DP, PP, PCP, TP).
3. Line 2168. TP groups are rows of the last dimension. No transpose is needed.
4. Lines 2173-2180. The comment says that only the TP group uses a MessageQueue broadcaster. The DCP group also asks for one, but it has one rank by default (vllm/distributed/parallel_state.py:2248-2254).

The other groups use the comment's recipe: move the dimension to the end, reshape, and unbind. PP uses transpose(2, 4) (vllm/distributed/parallel_state.py:2278-2280), DP uses transpose(1, 4) (vllm/distributed/parallel_state.py:2295). The EP group is only built for MoE models, and it spans DP × PCP × TP ranks of one PP stage:

    # Don't create EP group for dense models.
    if config.model_config is None or config.model_config.is_moe:
        group_ranks = (
            all_ranks.transpose(1, 2)
            .reshape(
                -1,
                data_parallel_size
                * prefill_context_model_parallel_size
                * tensor_model_parallel_size,
            )
            .unbind(0)
        )

An EPLB group with the same ranks follows only with --enable-eplb. The comment explains the separate group: EPLB weight transfers must not deadlock with the MoE collectives of the forward pass (vllm/distributed/parallel_state.py:2347-2368). At the end, every rank logs its position, for example "rank 5 in world size 8 is assigned as DP rank 1, …" (vllm/distributed/parallel_state.py:2372-2385).

Step 4: one GroupCoordinator per group#

By default, the constructor makes a PyTorch group for every row, so all ranks take part in every new_group call. Each rank keeps the pair for its own row: an NCCL device_group and a gloo cpu_group (vllm/distributed/parallel_state.py:501-518). Then it adds the optional parts:

        self.use_device_communicator = use_device_communicator
        self.device_communicator = None
        if use_device_communicator and self.world_size > 1:
            device_comm_cls = resolve_obj_by_qualname(
                current_platform.get_device_communicator_cls()
            )
            self.device_communicator = device_comm_cls(
                cpu_group=self.cpu_group,
                device=self.device,
                device_group=self.device_group,
                unique_name=self.unique_name,
                use_all2all=use_all2all,
            )

        from vllm.distributed.device_communicators.shm_broadcast import MessageQueue

        self.mq_broadcaster: MessageQueue | None = None
        if use_message_queue_broadcaster and self.world_size > 1:
            self.mq_broadcaster = MessageQueue.create_from_process_group(
                self.cpu_group, 1 << 22, 6
            )

        # TODO(#35915): Remove is_tpu() check once tpu_inference
        # overrides use_custom_op_collectives() to return True.
        self.use_custom_op_call = (
            current_platform.is_tpu() or current_platform.use_custom_op_collectives()
        )

1. Lines 549-560. The device communicator class comes from the platform: CudaCommunicator on CUDA and ROCm (vllm/platforms/cuda.py:581-584). A group with one rank gets none.
2. Lines 564-567. The shared-memory broadcaster has 6 chunks of 4 MiB (1 << 22). broadcast_object uses it when it exists (vllm/distributed/parallel_state.py:855-866).
3. Lines 571-574. On platforms that opt in, the collectives go through registered custom ops. CUDA opts in (vllm/platforms/cuda.py:707-709).

Step 5: the CUDA device communicator picks an all-reduce#

CudaCommunicator always makes a PyNcclCommunicator for a group with more than one rank (vllm/distributed/device_communicators/cuda_communicator.py:106-113). The faster kernels are only for TP groups:

        # Match the group name exactly so ETP does not enable TP-only backends.
        if unique_name.split(":")[0] != "tp":
            # custom allreduce or torch symm mem can be used only by tp
            use_custom_allreduce = False
            use_torch_symm_mem = False
            use_flashinfer_allreduce = False
            use_flashinfer_pcie_ipc_allreduce = False
            use_aiter_allreduce = False
        else:
            from vllm.distributed.parallel_state import _ENABLE_CUSTOM_ALL_REDUCE

            use_custom_allreduce = _ENABLE_CUSTOM_ALL_REDUCE
            use_torch_symm_mem = envs.VLLM_ALLREDUCE_USE_SYMM_MEM
            # FlashInfer all-reduce does not provide a fixed reduction order.
            use_flashinfer_allreduce = (
                envs.VLLM_ALLREDUCE_USE_FLASHINFER and not envs.VLLM_BATCH_INVARIANT
            )
            use_flashinfer_pcie_ipc_allreduce = (
                envs.VLLM_ALLREDUCE_USE_FLASHINFER_PCIE_IPC
                and not envs.VLLM_BATCH_INVARIANT
            )
            # Neither AITER nor QuickReduce all-reduce has a fixed reduction order.
            use_aiter_allreduce = (
                use_custom_allreduce
                and not envs.VLLM_BATCH_INVARIANT
                and bool(rocm_aiter_ops.is_custom_all_reduce_enabled())
            )

At run time, all_reduce tries the implementations in a fixed order and takes the first that accepts the tensor (vllm/distributed/device_communicators/cuda_communicator.py:362-446). On CUDA the chain has six links. They are NCCL symmetric memory (only with VLLM_USE_NCCL_SYMM_MEM=1), the FlashInfer PCIe all-reduce (opt-in), FlashInfer, the custom all-reduce, PyTorch symmetric memory, and pynccl. The size and dtype checks live in each implementation. The start-up log names the enabled ones for each group: "Using [...] all-reduce backends (in dispatch order) for group 'tp:0'" (vllm/distributed/device_communicators/cuda_communicator.py:353-360). The log puts NCCL symmetric memory after the two FlashInfer entries, but all_reduce tries it first (vllm/distributed/device_communicators/cuda_communicator.py:307-337, vllm/distributed/device_communicators/cuda_communicator.py:372-379).

Step 6: the collectives that the model calls#

Model code does not use the groups directly. It calls helpers such as tensor_model_parallel_all_reduce, which forward to the TP group (vllm/distributed/communication_op.py:12-21). The group then calls the registered op:

    def all_reduce(self, input_: torch.Tensor) -> torch.Tensor:
        """User-facing all-reduce function before we actually call the
        all-reduce operation.

        We need this because Dynamo does not support passing an arbitrary
        object (`self` in this case) to a custom op. We need to pass the
         group name as a string, and then look up the group coordinator from
         the group name, dispatch the all-reduce operation to the group
         coordinator.

        In addition, PyTorch custom ops do not support mutation or returning
        a new tensor in the same op. So we always make the all-reduce operation
        out-of-place.
        """
        # Bypass the function if we are using only 1 GPU.
        if self.world_size == 1:
            return input_

        if self.use_custom_op_call:
            return torch.ops.vllm.all_reduce(input_, group_name=self.unique_name)
        else:
            return self._all_reduce_out_place(input_)

    def _all_reduce_out_place(self, input_: torch.Tensor) -> torch.Tensor:
        if self.device_communicator is None:
            raise ValueError("No device communicator found")
        return self.device_communicator.all_reduce(input_)

1. Lines 740-741. With one rank, the call returns the input. No kernel runs.
2. Lines 743-744. torch.ops.vllm.all_reduce carries only the group name, a string. The docstring explains why: Dynamo cannot pass the Python object into a custom op (lines 726-737). The op looks the group up by name and calls the device communicator (vllm/distributed/parallel_state.py:199-204, vllm/distributed/parallel_state.py:399-403).
3. Lines 748-751. The op is out-of-place: it returns a new tensor.

all_gather and reduce_scatter work the same way (vllm/distributed/parallel_state.py:753-798). Pipeline stages use a different pair: isend_tensor_dict and irecv_tensor_dict on the PP group. The worker sends the IntermediateTensors without waiting, and the next stage receives them at the start of its step (vllm/v1/worker/gpu_worker.py:1336-1347, vllm/v1/worker/gpu_worker.py:1371-1378). broadcast_tensor_dict also exists, as a TP-group helper (vllm/distributed/communication_op.py:38-43); no code of Model Runner V2 calls it (derived from git grep).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

vllm serve <an MoE model> -tp 2 -pp 2 -dp 2      # one node, 8 GPUs

What can go wrong#

• Custom all-reduce turns itself off. On more than two PCIe-only GPUs, or when the P2P test fails, the log says "Custom allreduce is disabled …". The chain then falls through to symmetric memory or pynccl (vllm/distributed/device_communicators/custom_all_reduce.py:270-291). The server still works, but small all-reduces are slower.
• An unsupported TP size. TP = 3, 5 or 7 is not in the custom all-reduce list; vLLM warns and uses another implementation (vllm/distributed/device_communicators/custom_all_reduce.py:214-222).
• The wrong network interface. On a multi-node host, get_ip() may pick an interface that the other nodes cannot reach. Set VLLM_HOST_IP on each node (vllm/utils/network_utils.py:52-63).
• A rank that does not join. new_group is collective: every rank must make every group in the same order. If one worker fails before, the others wait until the PyTorch timeout. --distributed-timeout-seconds sets it (vllm/config/parallel.py:364-368, vllm/v1/worker/gpu_worker.py:1698-1700).
• Calling get_ep_group() for a dense model. It asserts, because the EP group exists only for MoE models (vllm/distributed/parallel_state.py:1659-1665).

Why it matters. Every multi-GPU feature of vLLM runs on these groups. When you know the rank layout, you can predict which GPUs talk in each collective, and which of them must sit on one node.