Pooling models and embeddings in vLLM serving

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

A generative model ends with an LM head and a sampler, and it runs many steps per request. A pooling model ends with a pooler. The pooler turns the hidden states of the prompt into one vector (an embedding), a few class scores, or one vector per token. The request then finishes after its prompt is processed: there is no decode phase.

vLLM runs one runner type per instance. With --runner pooling, the same engine core, scheduler and KV cache serve the request, but the worker calls the pooler instead of the sampler. vLLM can also convert a generative architecture: --convert embed or --convert classify wraps the model class. So, for example, a *ForCausalLM checkpoint can serve embeddings from its last hidden state.

An everyday comparison. A generative model is a writer who adds one word at a time. A pooling model is a reader who reads the whole text once and then writes one summary number or one index card. The comparison stops being true in one place. vLLM's reader may read a long text in several chunks, but only for some pooling types (Step 4).

The picture#

Read the panels from top to bottom. Amber is the generative path, teal and emerald the pooling path. A generative model never gets the pooling routes, and a pooling model never gets the chat routes.

In the code#

Step 1: the pooling tasks#

The task names are a Literal type. The old names score and encode are removed and raise a validation error with a hint:

PoolingTask = Literal[
    "embed",
    "classify",
    "token_embed",
    "token_classify",
    "plugin",
    "embed&token_classify",
]
POOLING_TASKS: tuple[PoolingTask, ...] = get_args(PoolingTask)

_REMOVED_POOLING_TASK_MESSAGES = {
    "score": "`score` task was removed; use `classify` instead.",
    "encode": (
        "`encode` task was removed; use `token_embed` or `token_classify` instead."
    ),
}


def check_removed_pooling_task(task: object) -> None:
    if isinstance(task, str) and (message := _REMOVED_POOLING_TASK_MESSAGES.get(task)):
        raise VLLMValidationError(message, parameter="task")


ScoreType = Literal["bi-encoder", "cross-encoder", "late-interaction"]
SCORE_TYPE_MAP: dict[PoolingTask, ScoreType] = {
    "embed": "bi-encoder",
    "classify": "cross-encoder",
    "token_embed": "late-interaction",
}

When a model supports several tasks, get_pooling_task picks one. The order is: --pooler-config '{"task": ...}' if set, else token_classify for *ForTokenClassification. Otherwise it takes the first of embed&token_classify, embed, classify, token_embed, token_classify, plugin (vllm/config/model.py:1918-1951). Reward models have no own task name; their default convert is embed (vllm/config/model.py:2353-2354).

Step 2: --runner and --convert#

Both options default to auto (vllm/config/model.py:163-169). The runner types are generate, pooling and draft; a pooling runner allows the converts embed and classify (vllm/config/scheduler.py:20, vllm/config/model.py:124-128).

• Runner. A set --convert forces pooling. Else a sentence-transformers config means pooling; else the registry decides; else the architecture suffix decides (vllm/config/model.py:1283-1329).
• Convert. A native pooling model needs none. Else the suffix decides, and a generative architecture run as pooling becomes embed (vllm/config/model.py:1331-1360).

_SUFFIX_TO_DEFAULTS: list[tuple[str, tuple[RunnerType, ConvertType]]] = [
    ("ForCausalLM", ("generate", "none")),
    ("ForConditionalGeneration", ("generate", "none")),
    ("ChatModel", ("generate", "none")),
    ("LMHeadModel", ("generate", "none")),
    ("ForTextEncoding", ("pooling", "embed")),
    ("EmbeddingModel", ("pooling", "embed")),
    ("ForSequenceClassification", ("pooling", "classify")),
    ("ForTokenClassification", ("pooling", "classify")),
    ("ForAudioClassification", ("pooling", "classify")),
    ("ForImageClassification", ("pooling", "classify")),
    ("ForVideoClassification", ("pooling", "classify")),
    ("ClassificationModel", ("pooling", "classify")),
    ("ForRewardModeling", ("pooling", "embed")),
    ("RewardModel", ("pooling", "embed")),
    # Let other `*Model`s take priority
    ("Model", ("pooling", "embed")),
]

The convert is applied when vLLM resolves the model class (vllm/model_executor/model_loader/utils.py:325-333). as_embedding_model subclasses the model and gives it an embedding pooler. By default, it takes the normalized hidden state of the last token (vllm/model_executor/models/adapters.py:250-281). as_seq_cls_model adds a score linear layer with num_labels outputs and a classification pooler (vllm/model_executor/models/adapters.py:307-378). Both return a class unchanged if it already pools.

vllm serve Qwen/Qwen3-Embedding-0.6B --runner pooling
# LAST-token embeddings from a chat model:
vllm serve meta-llama/Llama-3.1-8B-Instruct --convert embed

Step 3: Pooler and PoolerConfig#

Every pooler implements one small interface:

class Pooler(nn.Module, ABC):
    """The interface required for all poolers used in pooling models in vLLM."""

    @abstractmethod
    def get_supported_tasks(self) -> Set[PoolingTask]:
        """Determine which pooling tasks are supported."""
        raise NotImplementedError

    def get_pooling_updates(self, task: PoolingTask) -> PoolingParamsUpdate:
        """Construct the updated pooling parameters to use for a supported task."""
        return PoolingParamsUpdate()

    @abstractmethod
    def forward(
        self,
        hidden_states: torch.Tensor,
        pooling_metadata: PoolingMetadata,
    ) -> PoolerOutput:
        raise NotImplementedError

DispatchPooler holds one sub-pooler per task and calls the one for the request's task (vllm/model_executor/layers/pooler/special.py:25-57). --pooler-config sets PoolerConfig:

• Pooling types. Sequence pooling CLS, LAST, MEAN; token pooling ALL, STEP (vllm/config/pooler.py:16-20). pooling_type fills one of the two by its value (vllm/config/pooler.py:143-168).
• Defaults. The user's value wins. Next comes the sentence-transformers config of the checkpoint, then the model class default (vllm/config/model.py:775-803). The class defaults are LAST and ALL (vllm/model_executor/models/interfaces_base.py:163, vllm/model_executor/models/interfaces_base.py:173).
• Normalization. normalize was removed; use_activation replaces it (vllm/config/pooler.py:125-137). For embed, an unset use_activation means true: the embedding is L2-normalized (vllm/pooling_params.py:215-217).
• Dimensions. A request can ask for fewer dimensions only if the model is matryoshka; the head then cuts the vector before it normalizes (vllm/pooling_params.py:219-242, vllm/model_executor/layers/pooler/seqwise/heads.py:84-115).
• Long inputs. enable_chunked_processing and max_embed_len let an embedding model accept inputs above its position limit, by weighted averaging of chunks; not for multimodal inputs (vllm/config/pooler.py:80-96).

Step 4: what is turned off for pooling models#

• Async scheduling is off by default for pooling models. The code says it "negatively impacts performance of pooling models" (vllm/config/vllm.py:1664-1675). An explicit --async-scheduling is not rejected.
• Chunked prefill and prefix caching. Encoder-only (bidirectional) pooling models support neither. Decoder pooling models support both only with LAST or ALL pooling; MEAN, CLS and STEP turn them off (vllm/config/model.py:2193-2228, vllm/config/model.py:2239-2270). These properties set only the defaults. An explicit --enable-chunked-prefill or --enable-prefix-caching only logs a warning that the model does not officially support it (vllm/engine/arg_utils.py:3031-3035, vllm/engine/arg_utils.py:3051-3060, vllm/engine/arg_utils.py:3069-3078).
• Logits processors. --logits-processors on a pooling model raises ValueError; entry-point processors are skipped (vllm/v1/worker/gpu/sample/logits_processor/loader.py:172-179).
• The sampler. The model runner builds a PoolingRunner instead of a sampler (vllm/v1/worker/gpu/model_runner.py:467, vllm/v1/worker/gpu/model_runner.py:534-535).

Step 5: the endpoints#

The API server adds the pooling routes only when the model has a pooling task (vllm/entrypoints/launchers/api_server/routers.py:80-83). Which routes it adds depends on the task:

def register_pooling_api_routers(
    app: FastAPI,
    supported_tasks: tuple["SupportedTask", ...],
    model_config: ModelConfig | None = None,
):
    if model_config is None:
        return

    pooling_task = model_config.get_pooling_task(supported_tasks)

    if pooling_task is not None:
        from .pooling.api_router import router as pooling_router

        app.include_router(pooling_router)

    if "classify" in supported_tasks:
        from .classify.api_router import (
            router as classify_router,
        )

        app.include_router(classify_router)

    if pooling_task == "token_embed":
        from .token_embed.api_router import router as token_embed_router

        app.include_router(token_embed_router)
    elif "embed" in supported_tasks:
        from .embed.api_router import router as embed_router

        app.include_router(embed_router)

    if enable_scoring_api(supported_tasks, model_config):
        from .scoring.api_router import router as score_router

The serving classes send the prompt with PoolingParams through AsyncLLM.encode (vllm/v1/engine/async_llm.py:1034-1037). An IO processor plugin turns custom input into pooling input, and the result into a custom response. Only the pooling entry points use it (Plugins in the API server, Step 2).

Step 6: the pooling runner in Model Runner V2#

In a pooling model, the worker calls model_runner.pool() right after execute_model, not sample_tokens (vllm/v1/worker/gpu_worker.py:1355-1359). pool() calls the PoolingRunner and returns an AsyncPoolingOutput, which copies the result to the CPU on a side stream (vllm/v1/worker/gpu/model_runner.py:2318-2358):

    def pool(
        self,
        hidden_states: torch.Tensor,
        input_batch: InputBatch,
        req_states: RequestState,
    ) -> tuple[PoolerOutput, list[bool]]:
        hidden_states = hidden_states[: input_batch.num_tokens]
        pooling_metadata = self._get_pooling_metadata(
            input_batch, req_states, hidden_states.device
        )
        num_reqs = input_batch.num_reqs
        # Pooling has no speculative tokens, so this CPU upper bound is exact.
        seq_lens_cpu = input_batch.seq_lens_cpu_upper_bound[:num_reqs]
        pooling_metadata.build_pooling_cursor(
            input_batch.num_scheduled_tokens,
            seq_lens_cpu,
            device=hidden_states.device,
            query_start_loc_gpu=input_batch.query_start_loc[: num_reqs + 1],
        )
        pooler_output = self.model.pooler(hidden_states, pooling_metadata)
        finished_mask = pooling_metadata.get_pooling_cursor().get_finished_mask()
        pooler_output = self.late_interaction_runner.postprocess_pooler_output(
            raw_pooler_output=pooler_output,
            pooling_params=pooling_metadata.pooling_params,
            req_ids=input_batch.req_ids,
            finished_mask=finished_mask,
        )
        return pooler_output, finished_mask

The pooling cursor knows, per request, which rows of this step belong to it and whether the prompt is complete (finished_mask). The scheduler then finishes a request as soon as its pooler output arrives:

            elif request.pooling_params and pooler_output is not None:
                # Pooling stops as soon as there is output.
                request.status = RequestStatus.FINISHED_STOPPED
                stopped = True

Model Runner V2 supports the tasks embed, classify, token_embed, token_classify and embed&token_classify. A model whose selected task is not in this list fails at start-up with a hint to use VLLM_USE_V2_MODEL_RUNNER=0 (vllm/v1/worker/gpu/pool/pooling_runner.py:20-50).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

What can go wrong#

• 404 on /v1/chat/completions. The model runs as a pooling model, so the generate routes do not exist (vllm/entrypoints/launchers/api_server/routers.py:55-60).
• No /score or /rerank for a classifier. Scoring needs num_labels == 1 for classify (vllm/entrypoints/pooling/utils.py:231-236).
• "score task was removed". Use classify; encode became token_embed or token_classify (vllm/tasks.py:20-30).
• "Parameter normalize was removed". Use use_activation (vllm/config/pooler.py:125-135).
• "does not support Matryoshka embeddings". The request sent dimensions to a model without matryoshka support (vllm/pooling_params.py:225-230).
• Start-up error with a hint to VLLM_USE_V2_MODEL_RUNNER=0. The selected task is not one that Model Runner V2 supports (Step 6).

Why it matters. Embedding and reranking servers depend on settings that differ from chat serving. The pooling type decides whether chunked prefill and prefix caching work. The task decides which routes a client can call.