Tour: Life of a plugin

1pip install writes the entry points#

We are not in vLLM yet. The installer reads the package description and writes the entry points into the entry-point metadata of each package. The platform package publishes two entry points in two plugin groups:

setup(
    name="vllm_add_dummy_platform",
    version="0.1",
    packages=["vllm_add_dummy_platform"],
    entry_points={
        "vllm.platform_plugins": [
            "dummy_platform_plugin = vllm_add_dummy_platform:dummy_platform_plugin"  # noqa
        ],
        "vllm.general_plugins": [
            "dummy_custom_ops = vllm_add_dummy_platform:register_ops"
        ],
    },
)

The model package publishes one more general plugin, register_dummy_model (tests/plugins/vllm_add_dummy_model/setup.py:10-12). vLLM itself publishes two general plugins, the LoRA resolvers (pyproject.toml:45-47). No function of the packages runs at install time. The page Python entry points explains the format.

What the data looks like now (the general group across all installed packages):

[vllm.platform_plugins]
dummy_platform_plugin = vllm_add_dummy_platform:dummy_platform_plugin
[vllm.general_plugins]
dummy_custom_ops         = vllm_add_dummy_platform:register_ops
register_dummy_model     = vllm_add_dummy_model:register
lora_filesystem_resolver = vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
lora_hf_hub_resolver     = vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver

2vllm serve starts, and the CLI selects spawn#

The command vllm is a console script for vllm.entrypoints.cli.main:main (pyproject.toml:42-43). main() imports the module of each subcommand, then sets up the environment:

    import vllm.entrypoints.cli.benchmark.main
    import vllm.entrypoints.cli.collect_env
    import vllm.entrypoints.cli.download_kernels
    import vllm.entrypoints.cli.launch
    import vllm.entrypoints.cli.openai
    import vllm.entrypoints.cli.preload
    import vllm.entrypoints.cli.run_batch
    import vllm.entrypoints.cli.serve
    import vllm.entrypoints.cli.snapshot
    from vllm.entrypoints.common.startup import VLLM_SUBCMD_PARSER_EPILOG, cli_env_setup
    from vllm.utils.argparse_utils import FlexibleArgumentParser

    CMD_MODULES = [
        vllm.entrypoints.cli.openai,
        vllm.entrypoints.cli.serve,
        vllm.entrypoints.cli.launch,
        vllm.entrypoints.cli.benchmark.main,
        vllm.entrypoints.cli.collect_env,
        vllm.entrypoints.cli.download_kernels,
        vllm.entrypoints.cli.preload,
        vllm.entrypoints.cli.run_batch,
        vllm.entrypoints.cli.snapshot,
    ]

    if sys.argv[1:2] != ["snapshot"]:
        cli_env_setup()

cli_env_setup() sets VLLM_WORKER_MULTIPROC_METHOD=spawn when you did not set it (vllm/entrypoints/common/startup.py:147-149). This matters for the whole tour: each child process of vllm serve is a new Python interpreter (spawn). It has empty registries and no platform object, so it must load every plugin again. The page How vLLM finds and loads plugins explains fork and spawn.

3An import resolves current_platform: the dummy detector wins#

The imports of stop 2 reach a module-level from vllm.platforms import current_platform. One such line is in vllm/engine/arg_utils.py (vllm/engine/arg_utils.py:122), which the serve module imports through cli_args.py (vllm/entrypoints/cli/serve.py:13, vllm/entrypoints/common/cli_args.py:16). current_platform is not a real module attribute. So Python calls the module __getattr__, which resolves the platform on first use (vllm/platforms/__init__.py:303-322):

    platform_plugins = load_plugins_by_group(PLATFORM_PLUGINS_GROUP)

    activated_plugins = []

    for name, func in chain(builtin_platform_plugins.items(), platform_plugins.items()):
        try:
            assert callable(func)
            platform_cls_qualname = func()
            if platform_cls_qualname is not None:
                activated_plugins.append(name)
        except Exception:
            logger.debug(
                "Platform plugin %s failed during detection.",
                name,
                exc_info=True,
            )

    activated_builtin_plugins = list(
        set(activated_plugins) & set(builtin_platform_plugins.keys())
    )
    activated_oot_plugins = list(set(activated_plugins) & set(platform_plugins.keys()))

    if len(activated_oot_plugins) >= 2:
        raise RuntimeError(
            "Only one platform plugin can be activated, but got: "
            f"{activated_oot_plugins}"
        )
    elif len(activated_oot_plugins) == 1:
        platform_cls_qualname = platform_plugins[activated_oot_plugins[0]]()
        logger.info("Platform plugin %s is activated", activated_oot_plugins[0])

1. Line 250 loads the group vllm.platform_plugins. It imports dummy_platform_plugin, but does not call it yet.
2. Line 254 calls the five built-in detectors first, then the dummy detector. On a host with an NVIDIA GPU, the CUDA detector answers too.
3. Lines 277-279: exactly one out-of-tree detector is active, so it wins over CUDA. vLLM calls it a second time to get the class name, and logs "Platform plugin dummy_platform_plugin is activated" at INFO level.

The detector itself is short. A real detector checks for its device and returns None when the device is missing (docs/design/plugin_system.md:50). The test detector always answers:

def dummy_platform_plugin() -> str | None:
    # Real platform plugins often import vLLM modules while registering.
    from vllm.utils.torch_utils import direct_register_custom_op  # noqa: F401

    return "vllm_add_dummy_platform.dummy_platform.DummyPlatform"

Line 319 of platforms/__init__.py imports the class and makes one object. Its identity fields mark it as out-of-tree:

class DummyPlatform(Platform):
    _enum = PlatformEnum.OOT
    device_name = "DummyDevice"
    device_type: str = "privateuseone"
    dispatch_key: str = "PrivateUse1"

What the data looks like now (front end, NVIDIA host):

activated_plugins = ["cuda", "dummy_platform_plugin"]     # order of line 254
vllm.platforms._current_platform = DummyPlatform()        # is_out_of_tree() is True
plugins_loaded = False                                    # no general plugin has run yet

The platform is fixed before any general plugin runs. So a general plugin cannot change the platform choice. The page Platform plugins and current_platform explains the detection rules.

4The general plugins run for the first time#

The CLI builds the parsers of all subcommands before it reads your flags (vllm/entrypoints/cli/main.py:100-110). The serve parser calls make_arg_parser (vllm/entrypoints/cli/serve.py:172), which calls AsyncEngineArgs.add_cli_args (vllm/entrypoints/common/cli_args.py:466). That function loads the general plugins first:

    def add_cli_args(
        parser: FlexibleArgumentParser, async_args_only: bool = False
    ) -> FlexibleArgumentParser:
        # Initialize plugin to update the parser, for example, The plugin may
        # add a new kind of quantization method to --quantization argument or
        # a new device to --device argument.
        load_general_plugins()

load_general_plugins() sets the once-per-process flag, then calls each function of the group, in the order of entry_points() (vllm/plugins/__init__.py:82-90). Two of the four functions belong to our package:

• register_ops imports the module with the replacement layer (tests/plugins/vllm_add_dummy_platform/vllm_add_dummy_platform/__init__.py:12-13). The import runs the decorator register_oot:

# Register CustomRotaryEmbedding to CustomOP.
@RotaryEmbedding.register_oot
class DummyRotaryEmbedding(RotaryEmbedding):
    """Original rotary positional embedding."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.addition_config = True

    def forward_oot(self, *args, **kwargs) -> tuple[torch.Tensor, torch.Tensor]:
        return super().forward_oot(*args, **kwargs)

• register registers four architectures in the model registry. MyOPTForCausalLM uses the class form, so the function imports my_opt at once (tests/plugins/vllm_add_dummy_model/vllm_add_dummy_model/__init__.py:9-12). MyLlava uses the string form, a lazy registration (tests/plugins/vllm_add_dummy_model/vllm_add_dummy_model/__init__.py:26-27).

The two LoRA resolver functions also run. They register a resolver only when their environment variables are set (Plugins in the API server).

What the data looks like now (front end):

plugins_loaded = True
op_registry_oot = {"RotaryEmbedding": DummyRotaryEmbedding}           # custom_op.py:22
ModelRegistry.models["MyOPTForCausalLM"] = _RegisteredModel(...)      # model info computed now
ModelRegistry.models["MyLlava"] = _LazyRegisteredModel("vllm_add_dummy_model.my_llava", "MyLlava")

The key of op_registry_oot is the class name of the in-tree layer, because the decorator was called on RotaryEmbedding (vllm/model_executor/custom_op.py:337-340). The model info of an eager entry is computed at registration (vllm/model_executor/models/registry.py:974-979).

5pre_register_and_update runs four times#

This is the first platform hook of the start-up. The first call is at the end of add_cli_args, with the parser: current_platform.pre_register_and_update(parser) (vllm/engine/arg_utils.py:3257). The docstring names one use: an out-of-tree platform can import and register its own quantization config here (vllm/platforms/interface.py:642-650).

The CLI also builds the parsers of vllm launch render and vllm run-batch before it parses the flags. Both parsers call AsyncEngineArgs.add_cli_args too (vllm/entrypoints/cli/launch.py:99-106, vllm/entrypoints/cli/launch.py:39, vllm/entrypoints/cli/run_batch.py:62, vllm/entrypoints/launchers/run_batch.py:294-296). So the hook runs three times with a parser, once for each parser. (Derived from the parser loop of stop 4.) The general plugins do not run again in these calls, because the flag is already set.

The fourth call comes after the flags are parsed. build_async_engine_client_from_engine_args calls create_engine_config (vllm/entrypoints/launchers/api_server/entry.py:82), which starts like this:

        logging_config = self.create_logging_config()
        configure_logging_if_needed(logging_config)

        current_platform.pre_register_and_update()

        device_config = DeviceConfig(device=cast(Device, current_platform.device_type))

        current_platform.validate_environ(self.fail_on_environ_validation)

DummyPlatform does not override this hook, so the base method runs, and it does nothing (vllm/platforms/interface.py:638-651). Line 2248 makes the device config from device_type, which is "privateuseone" for the dummy platform. The docstring says that the hook runs "before ... cli arguments are parsed" (vllm/platforms/interface.py:644-645). That is true for the three calls with a parser, but not for the fourth call. So a real hook must give the same result when it runs again.

6ModelConfig asks the model registry#

Next, create_engine_config makes the model config (vllm/engine/arg_utils.py:2269). ModelConfig.__post_init__ reads the architecture name and asks the registry what the class can do (vllm/config/model.py:703-706). Then it resolves the architecture:

        model_info, arch = registry.inspect_model_cls(architectures, self)
        self._model_info = model_info
        self._architecture = arch
        logger.info("Resolved architecture: %s", arch)

The architecture is MyOPTForCausalLM, from --hf-overrides. The front end registered it at stop 4 with the class form, so the registry already has its model info. The eager entry returns it with no new process (vllm/model_executor/models/registry.py:981-982). The log shows "Resolved architecture: MyOPTForCausalLM" at INFO level.

7VllmConfig.__post_init__: the platform config hooks#

At the end, create_engine_config builds one VllmConfig from all parts (vllm/engine/arg_utils.py:2870). Its __post_init__ calls two more platform hooks. First, apply_config_platform_defaults runs (vllm/config/vllm.py:1804); the dummy platform keeps the base method, which does nothing (vllm/platforms/interface.py:653-664). Then vLLM adds the base mode of the CustomOp list:

        if all(s not in self.compilation_config.custom_ops for s in ("all", "none")):
            if (
                self.compilation_config.backend == "inductor"
                and self.compilation_config.mode != CompilationMode.NONE
            ):
                self.compilation_config.custom_ops.append("none")
            else:
                self.compilation_config.custom_ops.append("all")

With the default compilation (mode VLLM_COMPILE, backend inductor), the list becomes ["none"]: all custom ops are off. Later, the platform gets the last word (vllm/config/vllm.py:2046). The dummy platform turns all ops on:

    @classmethod
    def check_and_update_config(cls, vllm_config: VllmConfig) -> None:
        vllm_config.compilation_config.custom_ops = ["all"]

A real platform must also set the worker class here. CUDA shows the pattern:

    @classmethod
    def check_and_update_config(cls, vllm_config: VllmConfig) -> None:
        parallel_config = vllm_config.parallel_config
        model_config = vllm_config.model_config

        if parallel_config.worker_cls == "auto":
            parallel_config.worker_cls = "vllm.v1.worker.gpu_worker.Worker"

The default of worker_cls is "auto" (vllm/config/parallel.py:306). The dummy platform does not change it. That has a cost at stop 12.

What the data looks like now (the config that goes to the other processes):

vllm_config.compilation_config.custom_ops      = ["all"]   # dummy hook replaced ["none"]
vllm_config.parallel_config.worker_cls         = "auto"    # a real platform sets a class here
vllm_config.parallel_config.tensor_parallel_size = 2
vllm_config.parallel_config.distributed_executor_backend = "mp"   # world size 2

The executor backend comes from ParallelConfig.__post_init__: a world size above 1 gives "mp" when Ray is not in use (vllm/config/parallel.py:1055-1061, vllm/config/parallel.py:1096). The page The Platform interface lists every hook.

8AsyncLLM loads the stat loggers and starts the engine core#

The front end now makes AsyncLLM (vllm/entrypoints/launchers/api_server/entry.py:94). Its constructor loads one more plugin group:

        custom_stat_loggers = list(stat_loggers or [])
        custom_stat_loggers.extend(load_stat_logger_plugin_factories())

Our package has no stat logger, so the group is empty. The loader logs "No plugins for group ... found." at DEBUG level and returns (vllm/plugins/__init__.py:42-45). Then AsyncLLM starts the engine core (vllm/v1/engine/async_llm.py:193). The process manager uses the multiprocessing context, so the child is a spawned process:

        context = get_mp_context()
        common_kwargs = {
            "vllm_config": vllm_config,
            "local_client": local_client,
            "handshake_address": handshake_address,
            "executor_class": executor_class,
            "log_stats": log_stats,
            "tensor_queue": tensor_queue,
        }

            self.processes.append(
                context.Process(
                    target=EngineCoreProc.run_engine_core,
                    name=f"EngineCore_DP{global_index}" if is_dp else "EngineCore",
                    kwargs=common_kwargs
                    | {"dp_rank": global_index, "local_dp_rank": local_index},
                )

9The engine core detects the platform and runs the general plugins#

The spawned child imports the module of its target, vllm.v1.engine.core. On the way, Python runs the package file vllm/v1/engine/__init__.py, which imports vllm.config.kv_events (vllm/v1/engine/__init__.py:14). So Python runs the package vllm/config/__init__.py, which imports compilation.py, which has a module-level from vllm.platforms import current_platform (vllm/config/__init__.py:7, vllm/config/compilation.py:22). So stop 3 repeats in this process: the dummy detector runs twice more and wins again. Then EngineCore.__init__ runs:

        # plugins need to be loaded at the engine/scheduler level too
        from vllm.plugins import load_general_plugins

        load_general_plugins()

        self.vllm_config = vllm_config
        if not vllm_config.parallel_config.data_parallel_rank_local:
            logger.info(
                "Initializing a V1 LLM engine (v%s) with config: %s",
                VLLM_VERSION,
                vllm_config,
            )

        self.log_stats = log_stats
        # Opaque weight version supplied by the caller.
        self._weight_version = "default"

        # Setup Model.
        self.model_executor = executor_class(vllm_config)

Line 132 repeats stop 4 in this process. register_ops registers DummyRotaryEmbedding again, in this process's own op_registry_oot. register imports my_opt and computes its model info again. The engine core never builds the model, because the workers do. The eager form of the registration costs this import anyway. The lazy form would avoid it.

Line 147 makes the executor. This happens after the plugins, so the plugins are ready before any worker starts.

10The executor starts two worker processes#

The backend "mp" from stop 7 selects the multiproc executor (vllm/v1/executor/abstract.py:80-83). Its _init_executor starts one process for each local rank:

            for local_rank in range(self.local_world_size):
                global_rank = global_start_rank + local_rank
                is_driver_worker = self._is_driver_worker(global_rank)
                with cpu_omp_manager.configure_omp_envs(
                    rank=global_rank, local_rank=local_rank
                ):
                    unready_worker_handle = WorkerProc.make_worker_process(
                        vllm_config=self.vllm_config,
                        local_rank=local_rank,
                        rank=global_rank,
                        distributed_init_method=distributed_init_method,
                        input_shm_handle=scheduler_output_handle,
                        shared_worker_lock=shared_worker_lock,
                        is_driver_worker=is_driver_worker,
                        inherited_fds=inherited_fds,
                    )
                unready_workers.append(unready_worker_handle)

make_worker_process takes the same multiprocessing context, so each worker is spawned too (vllm/v1/executor/multiproc_executor.py:723). The target is WorkerProc.worker_main, and the pickled vllm_config goes with it (vllm/v1/executor/multiproc_executor.py:731-750). The page Executors and workers: from the engine core to the GPUs explains the executor.

11Each worker detects the platform, runs the plugins and the Triton hook#

Each worker child imports multiproc_executor.py to find its target. On the way, Python runs vllm/v1/executor/__init__.py, which imports abstract.py, which imports vllm.config (vllm/v1/executor/__init__.py:3, vllm/v1/executor/abstract.py:11). So the platform resolves in this process too. Then WorkerProc.__init__ calls init_worker (vllm/v1/executor/multiproc_executor.py:659-672):

        vllm_config.enable_trace_function_call_for_thread()

        from vllm.plugins import load_general_plugins

        load_general_plugins()

        # Let the platform replace core Triton kernels (e.g. CPU fallback
        # implementations) before any worker or model code can launch them.
        from vllm.platforms import current_platform

        current_platform.register_triton_kernel_overrides()

1. Line 284 repeats stop 4 a third and a fourth time, once in each worker. Now this worker has DummyRotaryEmbedding in its registry, before it builds any layer.
2. Line 290 is a kernel hook. A platform can replace core Triton kernels here, "before any worker or model code can launch them". The dummy platform keeps the base method, which does nothing (vllm/platforms/interface.py:375-386).

12The worker resolves worker_cls#

        parallel_config = vllm_config.parallel_config
        if isinstance(parallel_config.worker_cls, str):
            worker_class: type[WorkerBase] = resolve_obj_by_qualname(
                parallel_config.worker_cls
            )
        else:
            raise ValueError(
                "passing worker_cls is no longer supported. "
                "Please pass keep the class in a separate module "
                "and pass the qualified name of the class as a string."
            )

The worker class is the main hook of a hardware plugin: it owns the device, the model runner and the KV cache. Lines 345-347 of the same file then make the worker object inside set_current_vllm_config. Next, WorkerProc.__init__ calls init_device and load_model (vllm/v1/executor/multiproc_executor.py:680-688). With TP = 2, the device setup also asks the platform for its communicator class, because the tensor-parallel group has two ranks (vllm/distributed/parallel_state.py:549-552).

13The model is built, and attention asks the platform for a backend#

The loader finds the class in the registry of this worker (vllm/model_executor/model_loader/utils.py:305-313). The entry is there because of line 284 in stop 11. Then initialize_model calls the constructor inside set_current_vllm_config:

    signatures = inspect.signature(model_class.__init__)
    all_params = [param.name for param in signatures.parameters.values()]
    if "vllm_config" in all_params and "prefix" in all_params:
        # new-style model class
        with set_current_vllm_config(vllm_config, check_compile=True, prefix=prefix):
            model = model_class(vllm_config=vllm_config, prefix=prefix)
            record_metadata_for_reloading(model)
            return model

Each OPT decoder layer builds an OPTAttention, and its constructor builds an attention layer (vllm/model_executor/models/opt.py:137-144, vllm/model_executor/models/opt.py:106). The attention layer asks for its backend in __init__ (vllm/model_executor/layers/attention/attention.py:349-350). The selector asks the platform, a component picker:

@cache
def _cached_get_attn_backend(
    backend,
    attn_selector_config: AttentionSelectorConfig,
    num_heads: int | None = None,
    *,
    _run_kv_cache_dtype: CacheDType | None = None,
) -> type[AttentionBackend]:
    # Some platform selectors inspect the run-wide cache dtype. Keep it in the
    # cache key even though they read the value from the current config.
    from vllm.platforms import current_platform

    attention_cls = current_platform.get_attn_backend_cls(
        backend,
        attn_selector_config=attn_selector_config,
        num_heads=num_heads,
    )
    if not attention_cls:
        raise ValueError(
            f"Invalid attention backend for {current_platform.device_name}"
        )
    backend = resolve_obj_by_qualname(attention_cls)
    return backend

DummyPlatform.get_attn_backend_cls returns one class for every layer:

    @classmethod
    def get_attn_backend_cls(
        cls,
        selected_backend: "AttentionBackendEnum",
        attn_selector_config: "AttentionSelectorConfig",
        num_heads: int | None = None,
    ) -> str:
        return "vllm_add_dummy_platform.dummy_attention_backend.DummyAttentionBackend"  # noqa E501

Line 227 of the selector imports that class. @cache keeps one answer for each set of arguments. The arguments are the backend name, the AttentionSelectorConfig, num_heads and the KV cache data type (vllm/v1/attention/selector.py:158-180, vllm/v1/attention/selector.py:196-203). All OPT attention layers pass the same values, so each process asks the platform only once. (Derived from vllm/model_executor/models/opt.py:106-113.)

The test backend is a stub: it subclasses a placeholder backend and changes only its name (tests/plugins/vllm_add_dummy_platform/vllm_add_dummy_platform/dummy_attention_backend.py:7-10). At this commit, the file imports its parent class from vllm.attention.backends.placeholder_attn, a module that the repository does not contain (tests/plugins/vllm_add_dummy_platform/vllm_add_dummy_platform/dummy_attention_backend.py:4). A real vendor returns a backend that computes attention on its device. See How vLLM chooses an attention backend.

14CustomOp.__new__ swaps the replaced class#

The same constructors also build small layers such as norms and rotary embeddings. Each of them is a CustomOp. Before Python runs __init__, the class method __new__ looks at the op registry of out-of-tree classes:

    def __new__(cls, *args, **kwargs):
        try:
            op_name = cls.__name__
        except AttributeError:
            raise TypeError(
                f"Cannot instantiate '{cls.__name__}': its 'name' attribute "
                f"was not set, possibly because it was not decorated with "
                f"@CustomOp.register, or it's the CustomOp base class itself."
            ) from None

        if op_name not in op_registry_oot:
            op_cls_to_instantiate = cls
        else:
            op_cls_to_instantiate = op_registry_oot[op_name]
            logger.debug(
                "Instantiating custom op: %s using %s",
                op_name,
                str(op_cls_to_instantiate),
            )
        return super().__new__(op_cls_to_instantiate)

1. Line 109 takes the Python class name of the class that the model code builds, for example "RotaryEmbedding".
2. Lines 117-120: the name is in op_registry_oot, so vLLM makes the object from DummyRotaryEmbedding, and logs "Instantiating custom op" at DEBUG level.
3. Line 126 returns an object of the replacement class. It is a subclass of the in-tree class, so Python then runs its __init__, which sets addition_config = True.

For our model, the swap does not happen. OPT has learned position embeddings and builds no RotaryEmbedding (vllm/model_executor/models/opt.py:217). Its norm layer is LayerNorm (vllm/model_executor/models/opt.py:35, vllm/model_executor/models/opt.py:147). "LayerNorm" is not in op_registry_oot, so line 118 keeps the in-tree class. A model with standard RoPE, such as Llama, gets its rotary layer from get_rope, which builds RotaryEmbedding for the default rope type (vllm/model_executor/layers/rotary_embedding/__init__.py:147-155). There the swap happens. A test checks it (tests/plugins_tests/test_platform_plugins.py:51-64).

The lookup uses the exact class name. MRotaryEmbedding, which get_rope builds for an mrope_section, has another name, so it is not swapped (vllm/model_executor/layers/rotary_embedding/__init__.py:113-114). (Derived from line 109.)

15The op picks forward_oot, because the platform enabled it#

CustomOp.__init__ calls dispatch_forward once and keeps the result (vllm/model_executor/custom_op.py:128-134):

        enabled = self._enforce_enable or self.enabled()
        if enabled:
            compilation_config.enabled_custom_ops.update([self.__class__.name])
        else:
            compilation_config.disabled_custom_ops.update([self.__class__.name])

        if not enabled:
            # Compile forward_native to avoid eager torch ops if inside
            # opaque torch custom op (e.g. fused_moe, unified_attention, etc.)
            return self.maybe_compile(self.forward_native, enable=compile_native)

        if current_platform.is_rocm():
            return self.forward_hip
        elif current_platform.is_cpu():
            return self.forward_cpu
        elif current_platform.is_tpu():
            return self.forward_tpu
        elif current_platform.is_xpu():
            return self.forward_xpu
        elif current_platform.is_out_of_tree():
            return self.forward_oot
        else:
            return self.forward_cuda

The worker's copy of the config has custom_ops = ["all"] from stop 7. So the default enabled() returns true for every op (vllm/model_executor/custom_op.py:290, vllm/model_executor/custom_op.py:307). The platform is out-of-tree, so lines 202-203 select forward_oot. For OPT's LayerNorm, that is the default forward_oot, which calls forward_native (vllm/model_executor/custom_op.py:167-170). In a RoPE model, DummyRotaryEmbedding.forward_oot calls the parent's forward_oot (stop 4 excerpt, lines 18-19). RotaryEmbedding does not override that method, so the call ends in the same forward_native.

Without the dummy hook of stop 7, the list would be ["none"]. Then every op without a +name switch would be disabled and would use forward_native (line 192), and the vendor's forward_oot would not run for it. The page CustomOp, out-of-tree ops and IR op providers explains the switches.

16Block size, environment cache, and the server is ready#

After load_model, each worker finishes its start-up:

        # Set block size based on the attention backends
        current_platform.update_block_size_for_backend(vllm_config)

        # Initialize message queues after init_device() since multi-node setups
        # (nnodes_within_dp > 1) require distributed groups to be initialized
        self._init_message_queues(input_shm_handle, vllm_config)

        # Enable environment variable cache (e.g. assume no more
        # environment variable overrides after this point)
        enable_envs_cache()

1. Line 702 runs a config hook. It picks a block size that all attention backends of the model accept, when the user did not set one (vllm/platforms/interface.py:767-774, The Platform interface).
2. Line 710 freezes the environment variables. From now on, VLLM_PLUGINS and the other variables are read from a cache in this worker (vllm/envs.py:2308-2326).

The engine core does the same with enable_envs_cache() at the end of its own start-up (vllm/v1/engine/core.py:241-243). The front end then builds the HTTP app. build_app looks for endpoint plugins; our package has none (vllm/entrypoints/launchers/app.py:51). From the first request on, the forward method of each custom op calls the method that stop 15 chose (vllm/model_executor/custom_op.py:133-134). The tour Tour: Life of a chat request follows that request.