Tour: Build a toy plugin

1Lay out the package and publish the entry points#

A plugin package is a normal Python package. vLLM finds it only through its entry points. The toy has five modules:

vllm-toy-plugin/
  pyproject.toml
  vllm_toy_plugin/
    __init__.py       # register(): the general plugin (stop 2)
    model.py          # ToyLlamaForCausalLM (stop 3)
    ops.py            # ToyRMSNorm (stop 4)
    stat_logger.py    # ToyStatLogger (stop 6)
    endpoint.py       # ToyEndpointPlugin (stop 7)

The test plugins use setup.py with an entry_points dictionary. The stat logger test plugin is the shortest example:

setup(
    name="dummy_stat_logger",
    version="0.1",
    packages=["dummy_stat_logger"],
    entry_points={
        "vllm.stat_logger_plugins": [
            "dummy_stat_logger = dummy_stat_logger.dummy_stat_logger:DummyStatLogger"  # noqa
        ]
    },
)

vLLM itself uses the newer pyproject.toml form for its own general plugins:

[project.entry-points."vllm.general_plugins"]
lora_filesystem_resolver = "vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver"
lora_hf_hub_resolver = "vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver"

The toy uses the same form, with one table per plugin group:

# pyproject.toml (illustrative)
[build-system]
requires = ["setuptools>=64"]
build-backend = "setuptools.build_meta"

[project]
name = "vllm-toy-plugin"
version = "0.1.0"

[tool.setuptools]
packages = ["vllm_toy_plugin"]

[project.entry-points."vllm.general_plugins"]
toy_register = "vllm_toy_plugin:register"

[project.entry-points."vllm.stat_logger_plugins"]
toy_stat_logger = "vllm_toy_plugin.stat_logger:ToyStatLogger"

[project.entry-points."vllm.endpoint_plugins"]
toy_endpoint = "vllm_toy_plugin.endpoint:ToyEndpointPlugin"

The left side of each line is the entry-point name. VLLM_PLUGINS matches this name exactly, not the package name (vllm/plugins/__init__.py:63-64). The right side is module:attribute. Each group expects a different kind of object. The general group needs a function, the stat logger group a StatLoggerBase subclass, and the endpoint group a factory with no arguments. The table in How vLLM finds and loads plugins lists all six groups.

2Write the general plugin: register the model lazily#

vLLM calls a general plugin function with no arguments, once in each process (vllm/plugins/__init__.py:82-90). With TP = 2 that means four processes, and the function must do the same thing each time. The model test plugin shows the pattern: check first, then register. Its last two registrations use the lazy string form:

    # Test passing lazy model
    if "MyGemma2Embedding" not in ModelRegistry.get_supported_archs():
        ModelRegistry.register_model(
            "MyGemma2Embedding",
            "vllm_add_dummy_model.my_gemma_embedding:MyGemma2Embedding",
        )

    if "MyLlava" not in ModelRegistry.get_supported_archs():
        ModelRegistry.register_model("MyLlava", "vllm_add_dummy_model.my_llava:MyLlava")

The toy does the same for one architecture, and also imports the module with the op replacement:

# vllm_toy_plugin/__init__.py (illustrative)
def register():
    """General plugin. vLLM calls it once in every process."""
    from vllm import ModelRegistry

    # The model: a "module:Class" string. Nothing is imported now.
    if "ToyLlamaForCausalLM" not in ModelRegistry.get_supported_archs():
        ModelRegistry.register_model(
            "ToyLlamaForCausalLM", "vllm_toy_plugin.model:ToyLlamaForCausalLM"
        )

    # The op: importing the module runs @RMSNorm.register_oot (stop 4).
    import vllm_toy_plugin.ops  # noqa: F401

• from vllm import ModelRegistry is a lazy attribute of the vllm package (vllm/__init__.py:26).
• get_supported_archs() returns the keys of the registry (vllm/model_executor/models/registry.py:1157-1158).
• A string must contain exactly one colon. It becomes a lazy entry, and nothing is imported (vllm/model_executor/models/registry.py:1187-1193). A class would be inspected at once, in every process (vllm/model_executor/models/registry.py:1194-1195).
• The import of ops is inside the function. So the import, and the decorator, run only when vLLM calls the plugin. Python caches the module, so a second call in the same process does not register the op again.

The page Out-of-tree models and the model registry explains the registry and its lazy entries.

3Write the model class#

The model test plugin subclasses an in-tree model and changes one method:

class MyOPTForCausalLM(OPTForCausalLM):
    def compute_logits(self, hidden_states: torch.Tensor) -> torch.Tensor | None:
        # this dummy model always predicts the first token
        logits = super().compute_logits(hidden_states)
        if logits is not None:
            logits.zero_()
            logits[:, 0] += 1.0
        return logits

The toy subclasses Llama and changes nothing, so it runs any Llama-architecture checkpoint under a new name:

# vllm_toy_plugin/model.py (illustrative)
from vllm.model_executor.models.llama import LlamaForCausalLM


class ToyLlamaForCausalLM(LlamaForCausalLM):
    """Llama under a new architecture name. Override methods here."""

The class inherits the constructor __init__(self, *, vllm_config, prefix="", ...) (vllm/model_executor/models/llama.py:484-490). That matters: initialize_model checks for both parameter names, and otherwise warns and guesses old arguments (vllm/model_executor/model_loader/utils.py:96-115). The registry also checks the structure, for example embed_input_ids and compute_logits (vllm/model_executor/models/interfaces_base.py:69-76, vllm/model_executor/models/interfaces_base.py:140-147, Model capability protocols (SupportsPP, SupportsLoRA, ...)).

vLLM finds the class by the architecture name in the checkpoint's config.json. To use a normal Llama checkpoint, rename the architecture at start-up. A vLLM test does the same for its OPT model (tests/distributed/test_distributed_oot.py:20-21):

--hf-overrides '{"architectures": ["ToyLlamaForCausalLM"]}'

On the first start, the front end inspects the lazy entry in a subprocess and caches the result on disk (vllm/model_executor/models/registry.py:1088-1116). That subprocess runs register() too (vllm/model_executor/models/registry.py:1612). Later starts read the cache, until model.py changes.

4Replace an op with register_oot#

The platform test plugin replaces the rotary embedding layer. The decorator goes on a subclass of the in-tree class:

from vllm.model_executor.layers.rotary_embedding import RotaryEmbedding


# Register CustomRotaryEmbedding to CustomOP.
@RotaryEmbedding.register_oot
class DummyRotaryEmbedding(RotaryEmbedding):
    """Original rotary positional embedding."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.addition_config = True

    def forward_oot(self, *args, **kwargs) -> tuple[torch.Tensor, torch.Tensor]:
        return super().forward_oot(*args, **kwargs)

The toy replaces RMSNorm, which every Llama decoder layer builds (vllm/model_executor/models/llama.py:308-309):

# vllm_toy_plugin/ops.py (illustrative)
import torch

from vllm.model_executor.layers.layernorm import RMSNorm


@RMSNorm.register_oot            # the key is "RMSNorm", the class name
class ToyRMSNorm(RMSNorm):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.toy = True            # a visible mark: the swap happened

    def forward_cuda(self, x: torch.Tensor, residual: torch.Tensor | None = None):
        # Your CUDA kernel goes here. The toy calls the in-tree path.
        return super().forward_cuda(x, residual)

The decorator stores the class under a key and gives the class that key as its name:

    @classmethod
    def register_oot(cls, _decorated_op_cls=None, name: str | None = None):
        def decorator(op_cls):
            reg_name = name if name is not None else cls.__name__
            assert reg_name not in op_registry_oot, f"Duplicate op name: {reg_name}"
            op_cls.name = reg_name
            op_registry_oot[reg_name] = op_cls
            return op_cls

        if _decorated_op_cls is None:
            # Called with parentheses: @CustomOP.register_oot()
            # or @CustomOP.register_oot(name="...")
            # So, _decorated_op_cls is None.
            # We return the actual decorator function.
            return decorator
        elif isinstance(_decorated_op_cls, type):  # Check if it's a class
            # Called without parentheses: @CustomOP.register_oot
            # The first argument is the class itself.
            # We call the 'decorator' function immediately with the class.
            return decorator(_decorated_op_cls)
        else:
            # Handle other unexpected cases if necessary
            raise TypeError("Decorator can only be applied to classes.")

1. Line 337. Without name=, the key is the name of the class on which you call the decorator: RMSNorm.register_oot gives "RMSNorm".
2. Lines 338-340. A second registration with the same key fails the assert. The toy class gets name = "RMSNorm".
3. Lines 349-353. @RMSNorm.register_oot without parentheses passes the class as the first argument.
4. Lines 354-356. A string as the first argument raises TypeError.

When a Llama layer calls RMSNorm(...), CustomOp.__new__ finds "RMSNorm" in the out-of-tree registry and makes a ToyRMSNorm instead (vllm/model_executor/custom_op.py:107-126). The lookup uses the Python class name of the called class (line 109). So if you use the keyword form, the value must be the class name: @CustomOp.register_oot(name="RMSNorm"), not name="rms_norm". (Derived from lines 109 and 117.)

5Make the op actually run#

Each op picks one method when it is built. A disabled op takes forward_native; an enabled op takes the method of the platform (vllm/model_executor/custom_op.py:183-205). Whether an op is enabled depends on the list compilation_config.custom_ops:

        enabled = f"+{cls.name}" in custom_ops
        disabled = f"-{cls.name}" in custom_ops
        if enabled and disabled:
            raise ValueError(
                "custom_ops cannot both enable and disable the same operation: "
                f"{cls.name}. Remove either the '+' or '-' directive"
            )

        return (CustomOp.default_on() or enabled) and not disabled

The base mode is "all" or "none". vLLM adds it after it parses the flags:

        if all(s not in self.compilation_config.custom_ops for s in ("all", "none")):
            if (
                self.compilation_config.backend == "inductor"
                and self.compilation_config.mode != CompilationMode.NONE
            ):
                self.compilation_config.custom_ops.append("none")
            else:
                self.compilation_config.custom_ops.append("all")

With the default compilation (Inductor, mode VLLM_COMPILE), the base mode is "none". The page CustomOp, out-of-tree ops and IR op providers gives the full rule. So you have two switches:

The name quirk. The in-tree class registered the op name rms_norm (vllm/model_executor/layers/layernorm.py:38-39). But register_oot gave ToyRMSNorm the name "RMSNorm" (stop 4, line 339), and enabled() builds the switch from that name (line 282). So +rms_norm matches nothing in this model, and custom_op_log_check warns (vllm/config/compilation.py:1341-1361). (Derived from the cited lines; no test covers it.)

Which method runs on an enabled op depends on the platform. On CUDA it is forward_cuda (line 205 of custom_op.py). On an out-of-tree platform it is forward_oot (lines 202-203). The test platform turns every op on in its config hook (tests/plugins/vllm_add_dummy_platform/vllm_add_dummy_platform/dummy_platform.py:21-23). A general plugin such as the toy has no config hook, so the user must set the switch.

6Add a stat logger#

A stat logger receives the scheduler and iteration statistics in the front end. The test plugin implements the minimal interface:

class DummyStatLogger(StatLoggerBase):
    """A dummy stat logger for testing purposes.
    Implements the minimal interface expected by StatLoggerManager.
    """

    def __init__(self, vllm_config, engine_idx=0):
        self.vllm_config = vllm_config
        self.engine_idx = engine_idx
        self.recorded = []
        self.logged = False
        self.engine_initialized = False

    def record(
        self, scheduler_stats, iteration_stats, mm_cache_stats=None, engine_idx=0
    ):
        self.recorded.append(
            (scheduler_stats, iteration_stats, mm_cache_stats, engine_idx)
        )

    def log(self):
        self.logged = True

    def log_engine_initialized(self):
        self.engine_initialized = True

The base class has three abstract methods: the constructor, record and log_engine_initialized (vllm/v1/metrics/loggers.py:54-81). The toy counts the generated tokens and prints them:

# vllm_toy_plugin/stat_logger.py (illustrative)
from vllm.logger import init_logger
from vllm.v1.metrics.loggers import StatLoggerBase

logger = init_logger("vllm.toy_plugin")    # a child of "vllm", so INFO lines show


class ToyStatLogger(StatLoggerBase):
    def __init__(self, vllm_config, engine_index: int = 0):
        self.engine_index = engine_index
        self.generated = 0

    def record(self, scheduler_stats, iteration_stats,
               mm_cache_stats=None, engine_idx=0):
        if iteration_stats is not None:
            self.generated += iteration_stats.num_generation_tokens

    def log(self):
        logger.info("toy: engine %d generated %d tokens",
                    self.engine_index, self.generated)

    def log_engine_initialized(self):
        logger.info("toy: engine %d is ready", self.engine_index)

• The loader checks the class. An object that is not a StatLoggerBase subclass raises TypeError and stops start-up (vllm/v1/metrics/loggers.py:84-98).
• AsyncLLM.__init__ loads the group (vllm/v1/engine/async_llm.py:155-156). StatLoggerManager wraps a per-engine class in an adapter (vllm/v1/metrics/loggers.py:1408-1414). The adapter calls ToyStatLogger(vllm_config, engine_index) once for each engine (vllm/v1/metrics/loggers.py:442-445).
• record runs for each batch of engine outputs (vllm/v1/engine/async_llm.py:929-935). IterationStats has the field num_generation_tokens (vllm/v1/metrics/stats.py:483). The base class docstring warns that these stats classes are not a stable interface (vllm/v1/metrics/loggers.py:55-60).
• log runs every VLLM_LOG_STATS_INTERVAL seconds, default 10, when stats logging is on (vllm/entrypoints/launchers/utils/server_utils.py:78-84, vllm/envs.py:874-878).
• The logger name starts with vllm. on purpose. vLLM's logging config attaches its handler to the logger "vllm" only (vllm/logger.py:100-105). A logger with another name has no level of its own and gets the level of Python's root logger, which is WARNING by default. So Python drops its INFO lines. (Derived from the config and the Python logging defaults; other code can configure the root logger.)

7Add an endpoint#

An endpoint plugin adds HTTP routes. Its entry point gives a factory with no arguments, and the object must have name, required_tasks, attach_router and init_state (vllm/plugins/endpoint_plugins/interface.py:49-94). The test plugin uses the class itself as the factory:

class DummyAdminEndpointPlugin:
    name = "dummy_admin_endpoint_plugin"
    required_tasks: tuple[str, ...] | None = None

    def attach_router(self, app: FastAPI) -> None:
        @app.get("/v1/admin/scheduler_config")
        async def scheduler_config(raw_request: Request):
            engine_client = raw_request.app.state.dummy_engine_client
            if engine_client is None:
                raise HTTPException(
                    status_code=503,
                    detail="scheduler_config requires an engine, which this "
                    "server does not have",
                )
            results = await engine_client.collective_rpc("get_scheduler_config")
            return {"scheduler_config": results}

    async def init_state(self, engine_client, state, args) -> None:
        state.dummy_engine_client = engine_client

The toy route needs no engine, so its init_state does nothing:

# vllm_toy_plugin/endpoint.py (illustrative)
from fastapi import FastAPI


class ToyEndpointPlugin:
    name = "toy_endpoint"        # use the entry-point name here too
    required_tasks = None        # no task needed

    def attach_router(self, app: FastAPI) -> None:
        @app.get("/v1/toy/hello")
        async def hello():
            return {"plugin": "vllm_toy_plugin", "status": "loaded"}

    async def init_state(self, engine_client, state, args) -> None:
        pass                     # this route does not use the engine

1. Phase A. build_app calls attach_endpoint_plugins after the core routes (vllm/entrypoints/launchers/app.py:47-51). A plugin route with the same path and method as a core route replaces it (vllm/plugins/endpoint_plugins/interface.py:72-74).
2. Phase B. init_endpoint_plugins_state awaits init_state of each attached plugin (vllm/plugins/endpoint_plugins/interface.py:163-164).

The interface module says that an endpoint should not open new paths into the engine. A plugin that needs engine-side code pairs the endpoint with a general plugin (vllm/plugins/endpoint_plugins/interface.py:5-16). The page Plugins in the API server explains the route replacement in detail.

8Install and start the server#

CI installs a test plugin in editable mode, then runs the tests (.buildkite/test_areas/plugins.yaml:37-39). Do the same with the toy, in the same Python environment as vLLM:

pip install -e ./vllm-toy-plugin

# List EVERY plugin you need: the endpoint forces an allowlist (stop 7).
export VLLM_PLUGINS=toy_register,toy_stat_logger,toy_endpoint

vllm serve meta-llama/Llama-3.2-1B-Instruct \
    --hf-overrides '{"architectures": ["ToyLlamaForCausalLM"]}' \
    --compilation-config '{"custom_ops": ["+RMSNorm"]}'

curl http://localhost:8000/v1/toy/hello

The checkpoint is an example: any Llama-architecture checkpoint works. Three rules about VLLM_PLUGINS matter here:

• The value is split at commas, and the parts are not stripped (vllm/envs.py:1163-1167). Do not put spaces after the commas.
• The list also filters the platform group and vLLM's own general plugins. This list drops the two LoRA resolvers (pyproject.toml:45-47). Add lora_filesystem_resolver if you use it.
• An empty string is a list that matches nothing (vllm/plugins/__init__.py:107-109).

With TP = 2, register() runs in four processes; the stat logger and the endpoint load once, in the front end. The tour Tour: Life of a plugin shows where.

9Check that each part loaded#

Start with VLLM_LOGGING_LEVEL=DEBUG the first time. The default level is INFO (vllm/envs.py:860), and some of the lines below are DEBUG lines. The table lists what to look for, in start-up order:

You can also check the first two parts without a server, in one Python process. The functions below are the ones that vLLM itself calls; the test of the stat logger plugin uses the loader the same way (tests/plugins_tests/test_stats_logger_plugins.py:13-21):

# quick check (illustrative)
from vllm.plugins import load_general_plugins
from vllm.v1.metrics.loggers import load_stat_logger_plugin_factories

load_general_plugins()                       # runs toy_register once
from vllm import ModelRegistry
from vllm.model_executor.custom_op import op_registry_oot
print("ToyLlamaForCausalLM" in ModelRegistry.get_supported_archs())   # True
print(op_registry_oot)                       # {'RMSNorm': <class 'vllm_toy_plugin.ops.ToyRMSNorm'>}
print(load_stat_logger_plugin_factories())   # [<class 'vllm_toy_plugin.stat_logger.ToyStatLogger'>]