Multimodal serving: images, audio and video in vLLM

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

A vision-language model has a vision encoder next to the language model. The encoder turns an image into a sequence of vectors in the model's embedding space. The language model then reads them like token embeddings (The vision encoder↗). vLLM must fit this into its token-based scheduler and its KV cache.

The trick is the placeholder token. The multimodal processor of the model replaces the image marker in the prompt with N placeholder tokens, one per encoder output vector. From then on, the scheduler treats the request as a normal prompt of text plus N tokens. When a step reaches the placeholders, the worker runs the encoder and splices the N vectors into the input embeddings. The encoder cache keeps the output, keyed by a hash of the image. So a long prompt split over several steps needs no second encoder run, and nor does a second request with the same image.

An everyday comparison. Think of a book with a figure. The typesetter leaves a blank area of the right size on the page (the placeholders). A separate studio draws the figure (the encoder), and the printer pastes it into the blank area. The comparison stops being true in one place. vLLM keeps the drawn figure in a cache, and can paste it into other books with the same figure.

The picture#

Read the panels from top to bottom; each panel is one process. N depends on the model and the image size, so the figure leaves it as a symbol. The amber box is the one cache that spans two processes.

In the code#

Step 1: content parts and media loading#

A chat message can hold a list of content parts. The renderer has one parser per part type:

] = {
    "text": lambda part: _TextParser(part).get("text", None),
    "thinking": lambda part: _ThinkParser(part).get("thinking", None),
    "input_text": lambda part: _TextParser(part).get("text", None),
    "output_text": lambda part: _TextParser(part).get("text", None),
    "input_image": lambda part: _ResponsesInputImageParser(part).get("image_url", None),
    "image_url": lambda part: _ImageParser(part).get("image_url", {}).get("url", None),
    "image_embeds": lambda part: _ImageEmbedsParser(part).get("image_embeds", None),
    "audio_embeds": lambda part: _AudioEmbedsParser(part).get("audio_embeds", None),
    "video_embeds": lambda part: _VideoEmbedsParser(part).get("video_embeds", None),
    "prompt_embeds": lambda part: _PromptEmbedsParser(part).get("data", None),
    "image_pil": lambda part: _PILImageParser(part).get("image_pil", None),
    "audio_url": lambda part: _AudioParser(part).get("audio_url", {}).get("url", None),
    "input_audio": lambda part: _InputAudioParser(part).get("input_audio", None),
    "refusal": lambda part: _RefusalParser(part).get("refusal", None),
    "video_url": lambda part: _VideoParser(part).get("video_url", {}).get("url", None),
    "tool_reference": lambda part: cast(
        CustomChatCompletionContentToolReferenceParam, part
    ).get("name", None),

image_url, audio_url, input_audio and video_url load media. The image_embeds, audio_embeds and video_embeds parts take precomputed embeddings and need --enable-mm-embeds, which the config marks as "only for trusted users" (vllm/config/multimodal.py:326-338; the checks at vllm/renderers/chat_utils/content_parser.py:186-190, vllm/renderers/chat_utils/content_parser.py:214-218, vllm/renderers/chat_utils/content_parser.py:296-300). prompt_embeds needs --enable-prompt-embeds instead (vllm/renderers/chat_utils/content_parser.py:165-168). The renderer gets its media loader, the media connector, from a registry; VLLM_MEDIA_CONNECTOR picks it, default http (vllm/renderers/chat_utils/content_parser.py:140-151, vllm/envs.py:1072-1076).

    def load_from_url(
        self,
        url: str,
        media_io: MediaIO[_M],
        *,
        fetch_timeout: int | None = None,
    ) -> _M:  # type: ignore[type-var]
        if url[:5].lower() == "data:":
            return self._load_data_url(url, media_io)

        url_spec = parse_url(url)

        if url_spec.scheme and url_spec.scheme.startswith("http"):
            self._assert_url_in_allowed_media_domains(url_spec)
            max_bytes = media_io.get_max_bytes()

            cached = self._get_cached_bytes(url)
            if cached is not None:
                return media_io.load_bytes(cached)

            connection = self.connection
            budget_kwargs = (
                {"request_budget": self.download_budget}
                if self.download_budget is not None
                else {}
            )
            try:
                data = connection.get_bytes(
                    url_spec.url,
                    timeout=fetch_timeout,
                    allow_redirects=envs.VLLM_MEDIA_URL_ALLOW_REDIRECTS,
                    max_bytes=max_bytes,
                    **budget_kwargs,
                )
            except Exception as e:
                wrapped = _wrap_media_fetch_error(url, e)
                if isinstance(wrapped, VLLMUnprocessableEntityError):
                    raise wrapped from e
                raise

            self._put_cached_bytes(url, data)
            return media_io.load_bytes(data)

        if url_spec.scheme == "file":
            return self._load_file_url(url_spec, media_io)

        msg = "The URL must be either a HTTP, data or file URL."
        raise ValueError(msg)

• data: URLs must be base64 (vllm/multimodal/media/connector.py:328-346).
• HTTP(S): with --allowed-media-domains set, other hosts raise ValueError (vllm/multimodal/media/connector.py:370-379). Timeouts are VLLM_IMAGE_FETCH_TIMEOUT 5 s, VLLM_VIDEO_FETCH_TIMEOUT 30 s and VLLM_AUDIO_FETCH_TIMEOUT 10 s; one download is at most VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB 256 MB; redirects are allowed by default (vllm/envs.py:955-994). The timeouts are for the first attempt. By default vLLM makes up to 3 attempts, and each new attempt has a 4 times longer timeout (vllm/connections.py:292-306, vllm/connections.py:30).
• file: works only with --allowed-local-media-path, and only for files under it (vllm/multimodal/media/connector.py:348-368). The config calls this "a security risk" (vllm/config/model.py:231-237).

Step 2: the multimodal processor and its placeholders#

Each multimodal model class registers its processor on the global MULTIMODAL_REGISTRY with a decorator (vllm/multimodal/__init__.py:7, vllm/multimodal/registry.py:82-113); Llava is one example (vllm/model_executor/models/llava.py:487-491). The registry builds the processor for the served model (vllm/multimodal/registry.py:150-166). Its apply method does the work:

    def apply(
        self,
        inputs: ProcessorInputs,
        timing_ctx: TimingContext,
    ) -> MultiModalInput:
        """Process multi-modal inputs to be used in vLLM.

        The main steps are:

        1. Perform MM-specific post-processing to the token inputs.
        2. Apply HF Processor on multi-modal data together with optional
           dummy text, outputting processed multi-modal data tensors.
        3. Find and update sequences in the token IDs with placeholder tokens.
           The number of placeholder tokens equals the feature size of the
           multi-modal data outputted by the multi-modal encoder.
        4. Extract information about the placeholder tokens from the
           processed token IDs.
        """
        mm_res = self._cached_apply_hf_processor(inputs, timing_ctx)

        with timing_ctx.record("apply_prompt_updates"):
            prompt_ids, mm_placeholders = self._maybe_apply_prompt_updates(
                mm_items=inputs.mm_data_items,
                mm_res=mm_res,
            )

        mm_placeholder_ranges = {
            modality: [item.to_range() for item in placeholders]
            for modality, placeholders in mm_placeholders.items()
        }

        return mm_input(
            prompt_token_ids=prompt_ids,
            mm_kwargs=mm_res.kwargs,
            mm_hashes=mm_res.hashes,
            mm_placeholders=mm_placeholder_ranges,
        )

The result holds the new prompt token ids, the processed tensors (mm_kwargs), one hash per item, and the placeholder ranges. A range is an offset and a length in the prompt. The count limit is --limit-mm-per-prompt: 999 per modality when unset, capped by what the model supports. A prompt above it gets a validation error, "At most ... may be provided in one prompt." (vllm/config/multimodal.py:695-707, vllm/multimodal/processing/context.py:608-626).

--mm-processor-kwargs passes options to the HF processor, for example {"num_crops": 4} for Phi-3-Vision (vllm/config/multimodal.py:343-351). A request can also send mm_processor_kwargs or media_io_kwargs, but the server rejects non-empty values unless it runs with --trust-request-mm-kwargs (vllm/entrypoints/generate/base/protocol.py:70-95).

Step 3: the processor cache between the API server and the engine core#

Processing an image can cost more than the HTTP request itself. The processor cache keeps processed items by hash. Its type depends on the deployment:

def _get_cache_type(
    vllm_config: VllmConfig,
) -> Literal[None, "processor_only", "lru", "shm"]:
    model_config = vllm_config.model_config
    if not model_config.supports_multimodal_inputs:
        return None

    # Check if the cache is disabled.
    mm_config = model_config.get_multimodal_config()
    if mm_config.mm_processor_cache_gb <= 0:
        return None

    # Check if IPC caching is supported.
    parallel_config = vllm_config.parallel_config
    is_ipc_supported = parallel_config._api_process_count == 1 and (
        parallel_config.data_parallel_size == 1
        or parallel_config.data_parallel_external_lb
    )

    if not is_ipc_supported:
        return "processor_only"

    mm_config = model_config.get_multimodal_config()
    return mm_config.mm_processor_cache_type

• lru (default, one API server): the API server keeps a "sender" cache and the engine core a mirrored "receiver" cache. On a hit, the API server sends the hash without the data, and the engine core fills it in (vllm/multimodal/cache/factories.py:49-87).
• processor_only: with more than one API server, or with data parallelism without an external load balancer, only the API server caches its own processing results.
• shm: a shared-memory FIFO that the workers read; objects up to 128 MiB each (vllm/config/multimodal.py:369-378).

The size is --mm-processor-cache-gb, 4 GiB by default, once per API server process and once per engine core (vllm/config/multimodal.py:357-368). The two mirrored caches can drift. The engine core then raises MultiModalCacheMissError, and the API server drops the stale entries so a retry sends the data again (vllm/multimodal/cache/base.py:33-52).

Step 4: the encoder cache and its budget in the scheduler#

The scheduler owns an EncoderCacheManager. It counts in encoder embeddings, not in bytes, and shares entries between requests with the same hash (vllm/v1/core/encoder_cache_manager.py:19-46). Its size and the per-step encoder budget come from compute_mm_encoder_budget:

    encoder_compute_budget = max(
        scheduler_config.max_num_encoder_input_tokens, max_tokens_per_mm_item
    )
    encoder_cache_size = max(
        scheduler_config.encoder_cache_size, max_tokens_per_mm_item
    )

Both base values are max_num_batched_tokens; they are not configurable (vllm/config/scheduler.py:145-157, vllm/config/scheduler.py:306-307). So by default the cache holds as many embeddings as one step holds tokens, or one largest item if that is bigger.

When the scheduler gives a request new tokens, it checks the multimodal items in that token window. It schedules an encoder input only if its output is not cached, there is budget, and the cache has room. Otherwise it stops the request's chunk just before the item (vllm/v1/core/sched/scheduler.py:2007-2032). The SchedulerOutput lists the scheduled inputs per request and the hashes that workers may drop (vllm/v1/core/sched/output.py:255, vllm/v1/core/sched/output.py:263).

Step 5: the encoder run in Model Runner V2#

The worker keeps the encoder outputs in a dict from hash to tensor (vllm/v1/worker/gpu/mm/encoder_cache.py:8-13). Before the forward pass, the model state runs the encoder and builds the input embeddings:

    def prepare_inputs_embeds(
        self,
        scheduled_encoder_inputs: dict[str, list[int]],
        input_batch: InputBatch,
        req_states: RequestState,
    ) -> torch.Tensor:
        # Use unpadded input_ids to match is_mm_embed size (num_tokens).
        # input_batch.input_ids may be padded for CUDA graphs.
        input_ids_unpadded = input_batch.input_ids[: input_batch.num_tokens]

        if self.supports_mm_inputs:
            self.execute_mm_encoder(scheduled_encoder_inputs)

            mm_embeds, is_mm_embed = super().gather_mm_embeddings(input_batch)
            if self.mm_pruner is not None and mm_embeds:
                # EVS: recompute mrope positions for pruned media.
                mm_embeds = self.mm_pruner.recompute(mm_embeds, input_batch, req_states)
                # We must flush the staged rope updates for prepare_inputs() to pick up.
                self.apply_staged_writes()

            inputs_embeds = self.encoder_runner.get_inputs_embeds(
                input_ids_unpadded, mm_embeds, is_mm_embed
            )

1. prepare_mm_inputs skips items that are already in the cache (vllm/v1/worker/gpu/mm/encoder_runner.py:81-108).
2. execute_mm_encoder groups the items by modality and calls the model's embed_multimodal, or a captured encoder CUDA graph (vllm/v1/worker/gpu/mm/encoder_runner.py:150-176).
3. gather_mm_embeddings takes, for each request, the part of each item that falls into this step's tokens, and a boolean mask is_mm_embed. A step with only decodes gathers nothing (except for realtime models). A missing cache entry raises RuntimeError (vllm/v1/worker/gpu/mm/encoder_runner.py:207-294).
4. get_inputs_embeds calls the model's embed_input_ids with the embeddings and the mask, and copies the result into the fixed buffer for CUDA graphs (vllm/v1/worker/gpu/mm/encoder_runner.py:296-308).

The worker drops entries when the scheduler lists them in free_encoder_mm_hashes (vllm/v1/worker/gpu/model_runner.py:1187-1190).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

What can go wrong#

• "At most N image(s) may be provided in one prompt." The request is above --limit-mm-per-prompt or the model's own limit (Step 2).
• "Cannot load local files without --allowed-local-media-path." A file: URL without the flag (vllm/multimodal/media/connector.py:353-357).
• Slow TTFT for large images. The encoder runs inside the step that first reaches the image. The scheduler may also cut the chunk before an item that does not fit the encoder budget (Step 4).
• Per-request mm_processor_kwargs rejected. Start the server with --trust-request-mm-kwargs only for trusted clients (Step 2).
• Host memory. The processor cache exists once per API server and once per engine core (Step 3).

Why it matters. A multimodal request costs more than its token count shows: media download, processing, an encoder run and cache space. Knowing which cache sits where helps you to size the server and to keep repeated images cheap.