Expert parallelism and MoE serving

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

A Mixture-of-Experts layer has many expert MLPs, and a router sends each token to its top-k experts (Mixture of Experts↗). With expert parallelism, each GPU holds only some experts, and tokens travel to the GPUs of their experts and back (Expert parallelism↗). This page shows how vLLM builds such a layer, how it chooses the communication and the kernels, and which flags change it.

vLLM splits the work of one MoE layer into four parts. The router picks the experts. The dispatch moves the tokens. The experts compute. The combine brings the weighted results back. A modular MoE kernel keeps dispatch and combine in one object and the experts in another, so any communication method can work with any expert kernel. The communication method is the all2all backend.

Without --enable-expert-parallel, an MoE model still runs on several GPUs, but each expert is split with tensor parallelism over all DP × TP ranks. With the flag, each rank owns whole experts. A large MoE deployment usually runs data parallel attention plus expert parallel MoE: -dp 8 -ep.

An everyday comparison. Think of a hospital with specialists on several floors. At the front desk (the router), each patient gets a list of two specialists. Porters (dispatch) take the patients to the right floors, the specialists see them (experts), and the porters bring the patients back with the reports (combine). The comparison stops here. In vLLM, every patient on every floor moves in the same time slot, so a floor with no patients still joins each slot.

The picture#

Follow the token ids. Rank 0's token t0 chose experts 1 and 5. So both ranks compute part of it, and the combine sums the two parts on rank 0. Panel 3 is the code view: the runner owns the layer; the kernel is "prepare/finalize + experts".

In the code#

Step 1: FusedMoEFactory builds the layer#

A model builds an MoE layer with FusedMoEFactory. It is a function, not a class, and it returns an MoERunner (vllm/model_executor/layers/fused_moe/layer.py:92-153). It builds the parts in this order:

1. The MoE parallel config from the current TP, DP and PCP sizes (vllm/model_executor/layers/fused_moe/layer.py:238-244).
2. The expert counts: global = model experts + redundant experts (vllm/model_executor/layers/fused_moe/layer.py:76-89).
3. An EPLB layer state, only with EPLB on. Then the experts must divide evenly over the EP ranks (vllm/model_executor/layers/fused_moe/layer.py:269-284).
4. An ExpertMapManager that knows which experts are local (vllm/model_executor/layers/fused_moe/layer.py:291-302).
5. The router, unless the model passes its own (vllm/model_executor/layers/fused_moe/layer.py:306-346).
6. A FusedMoEConfig, the RoutedExperts object that holds the expert weights, and the MoERunner (vllm/model_executor/layers/fused_moe/layer.py:361-470).

The runner calls the layer through a registered custom op: torch.ops.vllm.moe_forward, or moe_forward_shared with shared experts. So torch.compile sees one opaque node (vllm/model_executor/layers/fused_moe/runner/moe_runner.py:309-327, vllm/model_executor/layers/fused_moe/runner/moe_runner.py:198-214).

Step 2: does this layer use expert parallelism?#

        use_ep = (
            dp_size_ * pcp_size_ * tp_size_ > 1
            and vllm_parallel_config.enable_expert_parallel
        )

        dp_size = dp_size_
        dp_rank = get_dp_group().rank_in_group if dp_size > 1 else 0
        pcp_size = pcp_size_
        pcp_rank = get_pcp_group().rank_in_group if pcp_size > 1 else 0
        tp_size, tp_rank = FusedMoEParallelConfig.flatten_tp_across_dp_and_pcp(
            tp_size_, dp_size_, dp_rank, pcp_size_, pcp_rank
        )

        if not use_ep:
            return FusedMoEParallelConfig(
                tp_size=tp_size,
                tp_rank=tp_rank,
                pcp_size=pcp_size,
                pcp_rank=pcp_rank,
                dp_size=dp_size,
                dp_rank=dp_rank,
                ep_size=1,
                ep_rank=0,
                sp_size=sp_size_,
                use_ep=False,
                all2all_backend=vllm_parallel_config.all2all_backend,
                enable_eplb=vllm_parallel_config.enable_eplb,
            )
        # DP + EP / TP + EP / DP + TP + EP
        assert use_ep
        # In EP, each device owns a set of experts fully. There is no tensor
        # parallel update tp_size, tp_rank, ep_size and ep_rank to reflect that.
        ep_size = tp_size
        ep_rank = tp_rank
        return FusedMoEParallelConfig(
            tp_size=1,
            tp_rank=0,
            pcp_size=pcp_size,
            pcp_rank=pcp_rank,
            dp_size=dp_size,
            dp_rank=dp_rank,
            ep_size=ep_size,
            ep_rank=ep_rank,
            sp_size=sp_size_,
            use_ep=True,
            all2all_backend=vllm_parallel_config.all2all_backend,
            enable_eplb=vllm_parallel_config.enable_eplb,

1. Lines 1198-1201. EP is used only when there is more than one rank and --enable-expert-parallel is set. The flag is off by default (vllm/config/parallel.py:190-191).
2. Lines 1207-1209. The TP size of the MoE layer is "flattened" over DP and PCP: DP × PCP × TP ranks (vllm/model_executor/layers/fused_moe/config.py:1105-1114).
3. Lines 1211-1226. Without EP, the experts are split with TP over all those ranks, and ep_size is 1.
4. Lines 1230-1231. With EP, the flattened TP size becomes the EP size, and the MoE TP size becomes 1: each rank owns whole experts.

The docstring lists six cases. One is "TP = 2, DP(PCP) = 2 and EP = True … the experts are split between the 4 devices" (vllm/model_executor/layers/fused_moe/config.py:1187-1195). The process group behind it is the EP group, built for every MoE model (Process groups: how vLLM slices its ranks).

Step 3: which experts live on which rank#

--expert-placement-strategy is linear by default: with 4 experts and 2 ranks, rank 0 gets [0, 1] and rank 1 gets [2, 3]. round_robin gives [0, 2] and [1, 3] (vllm/config/parallel.py:210-219). The expert_map is a tensor of shape [global_num_experts] with the local index of each local expert and −1 for the others. It exists only when ep_size > 1 (vllm/model_executor/layers/fused_moe/expert_map_manager.py:152-180).

Step 4: the modular kernel: prepare/finalize plus experts#

The design comment in modular_kernel.py names the four parts: "[Router] → [Quantize-Dispatch] → [Permute-Experts-Unpermute] → [Combine]" (vllm/model_executor/layers/fused_moe/modular_kernel.py:51-84). Two abstract classes carry them:

• FusedMoEPrepareAndFinalize: quantize and dispatch the inputs, then combine the outputs. Dispatch and combine sit in one class, because they may use collectives that must match (vllm/model_executor/layers/fused_moe/modular_kernel.py:186-193).
• FusedMoEExperts: the expert GEMMs, with checks such as is_supported_config for the device, quantization, activation and parallel config (vllm/model_executor/layers/fused_moe/modular_kernel.py:487-526, vllm/model_executor/layers/fused_moe/modular_kernel.py:558-614).

FusedMoEKernel joins one of each; both must be "modular" or both "monolithic". A monolithic pair works on the router logits instead of top-k ids (vllm/model_executor/layers/fused_moe/modular_kernel.py:1678-1710).

Step 5: selecting the experts implementation#

Each quantization type has an "oracle" that picks the experts class (vllm/model_executor/layers/fused_moe/oracle/base.py:1-20). For BF16 weights on CUDA, the priority list is FlashInfer TRT-LLM, FlashInfer CUTLASS, Triton, batched Triton. On Hopper (compute capability 9.x) both FlashInfer entries move to the back, because they are slower there. With DP > 1 FlashInfer CUTLASS moves back too:

    if current_platform.is_rocm():
        _AVAILABLE_BACKENDS = [
            UnquantizedMoeBackend.AITER,
            UnquantizedMoeBackend.TRITON,
            UnquantizedMoeBackend.BATCHED_TRITON,
        ]
    elif current_platform.is_cuda():
        _AVAILABLE_BACKENDS = [
            UnquantizedMoeBackend.FLASHINFER_TRTLLM,
            UnquantizedMoeBackend.FLASHINFER_CUTLASS,
            UnquantizedMoeBackend.TRITON,
            UnquantizedMoeBackend.BATCHED_TRITON,
        ]

        # On Hopper (SM90), the FlashInfer unquantized MoE kernels are slower
        # than Triton, so prefer Triton by default.
        if current_platform.is_device_capability_family(90):
            _move_to_back(_AVAILABLE_BACKENDS, UnquantizedMoeBackend.FLASHINFER_TRTLLM)
            _move_to_back(_AVAILABLE_BACKENDS, UnquantizedMoeBackend.FLASHINFER_CUTLASS)

        # HACK: Qwen3.5 has crash with FLASHINFER_CUTLASS BF16 if DEP.
        # Updating the oracle querying logic is out of the scope of this
        # PR. Need to fix the kernel or update structure in follow up.
        if moe_config.moe_parallel_config.dp_size > 1:
            _move_to_back(_AVAILABLE_BACKENDS, UnquantizedMoeBackend.FLASHINFER_CUTLASS)

The oracle takes the first backend whose kernel class accepts the config, and logs "Using <backend> Unquantized MoE backend out of potential backends: [...]" (vllm/model_executor/layers/fused_moe/oracle/unquantized.py:263-268, vllm/model_executor/layers/fused_moe/oracle/unquantized.py:359-372). --moe-backend forces a choice; its default is auto (vllm/config/kernel.py:302-306). Then the oracle builds the prepare/finalize object and logs its class name too (vllm/model_executor/layers/fused_moe/oracle/unquantized.py:518-530). Kernel internals are out of scope for this site.

Step 6: selecting dispatch and combine: --all2all-backend#

--all2all-backend defaults to allgather_reducescatter (vllm/config/parallel.py:220-234). The removed names naive and pplx fall back to it with a warning (vllm/config/parallel.py:537-543). maybe_make_prepare_finalize maps the choice to a class:

def maybe_make_prepare_finalize(
    moe: FusedMoEConfig,
    quant_config: FusedMoEQuantConfig | None,
    routing_tables: tuple[torch.Tensor, torch.Tensor, torch.Tensor] | None = None,
    allow_new_interface: bool = False,
    use_monolithic: bool = False,
    all2all_manager: Any | None = None,
    input_dtype: torch.dtype | None = None,
) -> FusedMoEPrepareAndFinalize | None:
    if moe.moe_parallel_config.use_passthrough_all2all:
        # The experts dispatch and combine themselves, in every EP topology.
        return PassThroughPrepareAndFinalize()

    if not moe.moe_parallel_config.use_all2all_kernels:
        if not allow_new_interface:
            return None

        # Opt-in XPU batched path: reorganize tokens into E x T x K locally
        # (no all-to-all) so BatchedTritonExperts (moe_mmk TD) can run.
        if current_platform.is_xpu() and moe.moe_backend == "batched_triton":
            return BatchedPrepareAndFinalize(
                max_num_tokens=moe.max_num_tokens,
                num_local_experts=moe.num_local_experts,
                num_dispatchers=1,
                rank=moe.moe_parallel_config.ep_rank,
            )

        # For DP/TP case, fall back to naive P/F.
        if moe.moe_parallel_config.dp_size > 1:
            logger.info_once(
                "Detected DP deployment with no --enable-expert-parallel. "
                "Falling back to AllGather+ReduceScatter dispatch/combine."
            )
            if all2all_manager is None:
                all2all_manager = get_ep_all2all_manager()
            return make_moe_prepare_and_finalize_naive_dp_ep(
                is_sequence_parallel=moe.moe_parallel_config.is_sequence_parallel,
                num_dispatchers=all2all_manager.world_size,
                use_monolithic=use_monolithic,
            )
        else:
            return make_moe_prepare_and_finalize_no_dp_ep(use_monolithic)

    if all2all_manager is None:
        all2all_manager = get_ep_all2all_manager()

    prepare_finalize: FusedMoEPrepareAndFinalize | None = None

    if moe.use_deepep_ht_kernels:
        assert moe.dp_size == all2all_manager.dp_world_size

1. Lines 191-193. passthrough: the MoE backend dispatches and combines by itself. vLLM binds it automatically for such backends; the CLI does not offer it (vllm/engine/arg_utils.py:1148-1151, vllm/config/kernel.py:194-229).
2. Lines 195-223. No all2all kernels (no EP, or one rank). With DP > 1 the experts are TP-split over all DP ranks, so each rank still needs every rank's tokens. vLLM logs "Detected DP deployment with no --enable-expert-parallel" and uses the all-gather/reduce-scatter object. Without DP, a local object needs no communication.
3. Lines 225-230 and on. With EP, the code takes the all2all manager of the EP group's device communicator. Then it builds one class per backend: DeepEP high-throughput, low-latency and v2, MoonEP, MoRI, FlashInfer NVLink, all-gather/reduce-scatter, NIXL-EP (lines 232-420).

The managers themselves are made by the EP group's CudaCommunicator when use_all2all is true (vllm/distributed/device_communicators/cuda_communicator.py:202-283). The default manager is AgRsAll2AllManager: dispatch is an all-gather, combine a reduce-scatter, over the DP group or, with sequence-parallel MoE, the EP group (vllm/distributed/device_communicators/all2all.py:67-173).

Step 7: what DP + EP looks like for a large MoE#

vllm serve <large MoE model> -dp 8 --enable-expert-parallel      # one node, 8 GPUs, TP = 1

• Eight engine cores, each a DPEngineCoreProc with one worker (vllm/v1/engine/core.py:1534-1537). Each rank runs attention only for its own requests, with its own KV cache.
• The MoE layers use EP size 8: each GPU owns 1/8 of the experts. The EP group spans all 8 ranks (Process groups: how vLLM slices its ranks).
• All ranks step together in waves; an idle rank runs dummy batches, and the workers agree on the padded batch size each step (Data parallel serving: several engine cores behind one endpoint).
• With TP > 1, DP > 1 and EP, the MoE input is sequence parallel for most backends. So TP ranks do not compute the same tokens twice (vllm/config/parallel.py:759-785). This applies only to models that pass the setting to their MoE layer, for example Qwen3-MoE (vllm/model_executor/models/qwen3_moe.py:152, vllm/model_executor/layers/fused_moe/layer.py:59).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

What can go wrong#

• EP without DP or TP does nothing. With one rank, use_ep is false even with the flag (vllm/model_executor/layers/fused_moe/config.py:1198-1201).
• EPLB flags without EP. --enable-eplb without --enable-expert-parallel raises "enable_expert_parallel must be True to use EPLB." (vllm/config/parallel.py:569-570).
• Uneven experts. With EPLB, global experts must divide by the EP size (vllm/model_executor/layers/fused_moe/layer.py:272-279). Choose num_redundant_experts to fit.
• Missing library. DeepEP, NIXL-EP, MoRI or FlashInfer backends need their packages; an unknown backend name raises "Unknown all2all backend" (vllm/distributed/device_communicators/cuda_communicator.py:277-278).
• The default backend moves every token to every rank. allgather_reducescatter is simple but sends DP × the tokens of each rank in each layer (vllm/distributed/device_communicators/all2all.py:128-161). At scale, the DeepEP-style backends send only the needed tokens.
• Unexpected fallback runner. --enable-dbo moves the model to the fallback model runner unless VLLM_USE_V2_MODEL_RUNNER is set (DBO deep dive).

Why it matters. The largest open models are MoE models. Their throughput depends on how tokens move between GPUs, so the all2all backend, the expert kernel and EPLB are among the most important serving choices.