The OpenAI-compatible server: launch modes, app, serving classes and streaming

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

The page The OpenAI-compatible API shows the API from the outside: routes, fields and the stream format. This page shows the inside of the API server. The API server is one process (or several) with an asyncio event loop. uvicorn reads HTTP, FastAPI matches the route, and a route function calls a serving class. The serving class checks the request, asks the renderer for token ids, makes the sampling parameters and calls AsyncLLM, the EngineClient (The engine client: AsyncLLM in the API server).

The server makes one object per serving class at start-up and keeps it in the app state. So a route function is short: it takes the serving object from app.state, calls one method, and turns the result into an HTTP response. The result is an error, a full JSON body, or an async generator of SSE strings.

Some work is the same for every route. Middleware checks CORS and the API key. Exception handlers turn Python exceptions into HTTP status codes. Decorators on the route functions watch for a client disconnect and count the load.

An everyday comparison. Think of the front desk of a large office. A guard at the door checks the visitor's badge (middleware). Signs send the visitor to the right counter (the router). The clerk at the counter checks the form and copies it onto the internal form (renderer and sampling parameters). Then the clerk puts it into the internal mail (AsyncLLM). The comparison stops being true for streams. The clerk promises the answer piece by piece before the mail has left the building. So a stream starts with HTTP 200 before the engine sees the request.

The picture#

Read panel 1 along the arrows: the top row first, then the bottom row. Blue boxes are library code (uvicorn, Starlette, FastAPI); violet boxes are vLLM code; the amber box is the boundary to the engine. All six layers run in one process, on one event loop.

In the code#

Step 1: the launch modes of vllm serve#

ServeSubcommand.cmd picks one of five ways to start (vllm/entrypoints/cli/serve.py:54-156). The page entrypoints/cli/serve.py: the vllm serve subcommand and its launch modes reads it line by line; the short form is:

• One API server (the default when the DP size is 1). The CLI process itself runs uvloop.run(run_server(args)) (vllm/entrypoints/cli/serve.py:153-156). run_server binds the socket first and then builds the engine client and the app (vllm/entrypoints/launchers/api_server/entry.py:164-201). The comment gives the reason for the early bind: it avoids a race with Ray (vllm/entrypoints/launchers/launcher.py:372-379).
• Several API servers. --api-server-count defaults to the DP size with internal load balancing (vllm/entrypoints/cli/serve.py:113-130). Then the CLI process becomes a supervisor. It binds one socket with SO_REUSEPORT, starts the engine cores and spawns ApiServer_i processes (vllm/entrypoints/cli/serve.py:298, vllm/v1/utils.py:248-254). Each process runs the same app with its own AsyncLLM. A process that has more open connections than another one waits up to 5 ms before it accepts a new connection (vllm/entrypoints/launchers/launcher.py:103-125).
• Headless. --headless sets the count to 0 and starts only engine cores, for multi-node data parallelism (vllm/entrypoints/cli/serve.py:69-77, vllm/entrypoints/cli/serve.py:181-266).
• gRPC. --grpc starts a gRPC server on --port instead of HTTP. It wraps AsyncLLM with the VllmEngineServicer of the external smg-grpc-servicer package (vllm/entrypoints/cli/serve.py:59-63, vllm/entrypoints/launchers/grpc_server.py:56-85).

The Rust frontend (one paragraph). rust/ holds an experimental OpenAI-compatible frontend in Rust. It talks to the same engine core processes over the same ZMQ boundary (rust/README.md:1-5). With VLLM_USE_RUST_FRONTEND=1, vllm serve takes the multi-server path, forces the API server count to 1, and starts the Rust binary instead of Python API servers (vllm/entrypoints/cli/serve.py:131-136, vllm/entrypoints/cli/serve.py:346-365). The README calls it experimental and not feature-complete, and this site does not explain it further.

Step 2: build_app makes the FastAPI app#

After AsyncLLM is ready, build_and_serve asks the engine for the supported tasks, builds the app, fills the app state and starts uvicorn (vllm/entrypoints/launchers/api_server/entry.py:115-161):

    if args.disable_fastapi_docs:
        app = FastAPI(
            openapi_url=None, docs_url=None, redoc_url=None, lifespan=lifespan
        )
    elif args.enable_offline_docs:
        app = FastAPI(docs_url=None, redoc_url=None, lifespan=lifespan)
    else:
        app = FastAPI(lifespan=lifespan)
    app.state.args = args
    app.root_path = args.root_path

    register_api_routers(args, app, supported_tasks, model_config)

    # Endpoint plugin hooks run after core routers so plugins can inspect them.
    # This runs even for the CPU only render server. A plugin eligible for
    # the `render` task still gets its routes registered. It receives
    # `engine_client=None` at Phase B (see `_init_endpoint_plugins_state`).
    attach_endpoint_plugins(app, supported_tasks)

    init_exception_handler(app)
    init_entrypoints_middleware(args, app, supported_tasks)
    app = sagemaker_standards_bootstrap(app)
    return app

1. Lines 34-41: --disable-fastapi-docs removes /docs, /redoc and the OpenAPI schema; --enable-offline-docs serves the bundled Swagger files.
2. Line 45: register_api_routers adds the routers by task and flag (vllm/entrypoints/launchers/api_server/routers.py:12-90). The route table is on The OpenAI-compatible API.
3. Line 51: endpoint plugins attach their routes after the core routes, so they can replace one (Plugins in the API server).
4. Line 53: the exception handlers, in four layers (vllm/entrypoints/serve/exception_handling/register.py:26-42).
5. Line 54: the middleware (vllm/entrypoints/serve/middleware/register.py:19-75):
   • CORSMiddleware, always. --allowed-origins is empty by default, so browsers on other origins are refused (vllm/entrypoints/common/cli_args.py:318-325).
   • AuthenticationMiddleware, only with --api-key or VLLM_API_KEY. It compares SHA-256 hashes in constant time and guards only /v1, /v2, /inference and /cohere paths (vllm/entrypoints/serve/middleware/authenticate.py:11, vllm/entrypoints/serve/middleware/authenticate.py:28-45). All other paths need no key, also /invocations (see "What can go wrong").
   • XRequestIdMiddleware, only with --enable-request-id-headers. It copies the request's X-Request-Id, or a new UUID, into the response header (vllm/entrypoints/serve/middleware/x_request_id.py:26-36).
   • ScalingMiddleware for the generate task (elastic EP), and the classes or functions named by --middleware.
6. Line 55: the external package model_hosting_container_standards wraps the app for SageMaker hosting (vllm/entrypoints/serve/sagemaker/api_router.py:145-159). The SageMaker routes /ping and /invocations are already registered on every server (vllm/entrypoints/launchers/api_server/routers.py:40-44). /invocations picks a handler by the type of the body and runs the same serving objects as the /v1 routes (vllm/entrypoints/serve/sagemaker/api_router.py:98-132).

serve_http then prints every route and runs uvicorn with NoSignalServer, which leaves SIGINT and SIGTERM to vLLM's own shutdown handler (vllm/entrypoints/launchers/launcher.py:47-72, vllm/entrypoints/launchers/launcher.py:251-291). A watchdog task checks the engine every 5 seconds (vllm/entrypoints/launchers/launcher.py:314-337).

Step 3: init_app_state makes the serving objects#

    state.engine_client = engine_client
    state.log_stats = not args.disable_log_stats
    state.vllm_config = vllm_config
    state.args = args
    resolved_chat_template = load_chat_template(args.chat_template)
    default_chat_template_kwargs = resolve_default_chat_template_kwargs(args)
    state.tool_server = (
        await init_tool_server(args) if "generate" in supported_tasks else None
    )

    # Merge default_mm_loras into the static lora_modules
    default_mm_loras = (
        vllm_config.lora_config.default_mm_loras
        if vllm_config.lora_config is not None
        else {}
    )
    lora_modules = process_lora_modules(args.lora_modules, default_mm_loras)

    state.openai_serving_models = OpenAIServingModels(
        engine_client=engine_client,
        base_model_paths=base_model_paths,
        lora_modules=lora_modules,
    )
    await state.openai_serving_models.init_static_loras()

    state.online_renderer = OnlineRenderer(
        model_config=engine_client.model_config,
        renderer=engine_client.renderer,
        request_logger=request_logger,
        chat_template=resolved_chat_template,
        chat_template_content_format=args.chat_template_content_format,
        trust_request_chat_template=args.trust_request_chat_template,
        trust_request_mm_kwargs=args.trust_request_mm_kwargs,
        enable_auto_tools=args.enable_auto_tool_choice,
        exclude_tools_when_tool_choice_none=args.exclude_tools_when_tool_choice_none,
        tool_parser=args.tool_call_parser,
        tool_strict_level=args.tool_strict_level,
        reasoning_parser=args.structured_outputs_config.reasoning_parser,
        default_chat_template_kwargs=default_chat_template_kwargs,
        log_error_stack=args.log_error_stack,
    )
    state.online_renderer.warmup()

The state holds the engine client, the args, and the objects that every server needs. These are OpenAIServingModels (for /v1/models and LoRA names), the OnlineRenderer and the ServingTokenization object (vllm/entrypoints/launchers/api_server/app_state.py:122-130). For the generate task, init_generate_state adds one object per generate API (vllm/entrypoints/generate/api_router.py:104-172):

Each object reads its flags once, at construction. For example, OpenAIServingChat reads the model's default sampling parameters and builds its parser class in __init__ (vllm/entrypoints/openai/chat_completion/serving.py:167-187). So a change of the model's generation_config.json needs a restart.

Step 4: the serving class hierarchy#

The three layers split the work:

1. BaseServing knows the model names. _check_model accepts the served name, a loaded LoRA name, or (with VLLM_ALLOW_RUNTIME_LORA_UPDATING) a name that a LoRA resolver can load. Any other name gives HTTP 404 NotFoundError (vllm/entrypoints/serve/engine/serving.py:49-84). create_error_response maps an exception class to a status code (vllm/entrypoints/serve/exception_handling/error_response.py:14-96).
2. GenerateBaseServing knows the engine. It takes the renderer and the input processor from the engine client (vllm/entrypoints/generate/base/serving.py:148-151). Its _preflight runs the engine checks that must come before a stream starts:

   vllm/entrypoints/generate/base/serving.py:169-181GitHub ↗Copy

   169170171172173174175176177178179180181

       def _preflight(self, n: int = 1) -> None:
           """Engine checks that must run before a response is started.
   
           This is required for the streaming case, where we return a success
           status before we actually start generating text :).
   
           Args:
               n: Number of sequences the request will occupy.
   
           """
           if self.engine_client.errored:
               raise self.engine_client.dead_error
           self.engine_client.check_admission(n)

3. OpenAIServingChat (and its siblings) knows one API format: the fields, the chunks and the full response.

Step 5: the request flow inside one serving call#

For /v1/chat/completions, _create_chat_completion runs these steps in order (vllm/entrypoints/openai/chat_completion/serving.py:302-438). /v1/completions has the same shape without the chat template (vllm/entrypoints/openai/completion/serving.py:131-296).

1. Validate. Pydantic and the model validators of the request class have already run in FastAPI. Errors there give HTTP 400 (The OpenAI-compatible API, errors deep dive).
2. Check the model and the engine. _check_model, then _preflight(n) (vllm/entrypoints/openai/chat_completion/serving.py:278-283).
3. Render. The online renderer applies the chat template and tokenizes; the prompt length check runs here (vllm/entrypoints/openai/chat_completion/serving.py:285). See Renderers and chat templates: from messages to token ids.
4. Request id. "chatcmpl-" + the X-Request-Id header, else the body field request_id (a random UUID by default) (vllm/entrypoints/openai/chat_completion/serving.py:320-322, vllm/entrypoints/serve/engine/serving.py:132-142).
5. Sampling parameters. get_max_tokens, then to_sampling_params with output_kind DELTA for a stream and FINAL_ONLY otherwise (vllm/entrypoints/openai/chat_completion/serving.py:349-369).
6. Make the generator. engine_client.generate(...) with the priority, the DP rank header and the reasoning flags (vllm/entrypoints/openai/chat_completion/serving.py:396-408). This only makes an async generator; the engine sees nothing yet.
7. Answer. For a stream, return chat_completion_stream_generator; else await chat_completion_full_generator (vllm/entrypoints/openai/chat_completion/serving.py:415-438).

Step 2 and step 6 meet: check_admission runs in _preflight and again inside AsyncLLM.add_request, when the generator starts (vllm/v1/engine/async_llm.py:551, vllm/v1/engine/async_llm.py:575-578). The first check gives a clean HTTP 503 before a stream starts; the second one sees the requests that other tasks added in between.

Step 6: streaming#

The route wraps the generator of SSE strings in a StreamingResponse (vllm/entrypoints/openai/chat_completion/api_router.py:80-85). Three details matter in production:

• Status first. The HTTP status 200 goes out with the first bytes. Later errors become a data: error chunk, and data: [DONE] always ends the stream (vllm/entrypoints/openai/chat_completion/serving.py:485-508).
• Keep-alive comments. With --sse-keep-alive-interval N (default 0, off), with_sse_keep_alive sends : keep-alive after N idle seconds. Then a proxy does not close the connection during a long queue or prefill (vllm/entrypoints/serve/utils/sse_keep_alive.py:18-33, vllm/entrypoints/common/cli_args.py:156-162).
• Disconnect means abort. Before the route returns, with_cancellation runs the handler and a disconnect listener as two tasks; the first to finish cancels the other (vllm/entrypoints/serve/utils/api_utils.py:92-111). After the route returns a StreamingResponse, Starlette watches for the disconnect. Either way AsyncLLM.generate() gets the cancel and sends ABORT to the engine core (vllm/v1/engine/async_llm.py:807-812). Tour 1 follows this path in its cancellation deep dive (Tour: Life of a chat request).

Step 7: non-streaming answers, n > 1, logprobs and usage#

Without a stream, the output kind is FINAL_ONLY. The output processor makes no RequestOutput until the request is finished (vllm/v1/engine/output_processor.py:364-369). With n > 1, AsyncLLM sends n child requests, and the parent collects their final outputs. It returns one RequestOutput with all n outputs when the last child finishes (vllm/v1/engine/parallel_sampling.py:117-128). The full generator then reads one result:

        created_time = int(time.time())
        final_res: RequestOutput | None = None

        try:
            async for res in result_generator:
                final_res = res
        except asyncio.CancelledError:
            return self.create_error_response("Client disconnected")

        if final_res is None:
            return self.create_error_response(
                "No output received from the engine.",
                err_type="InternalServerError",
                status_code=HTTPStatus.INTERNAL_SERVER_ERROR,
            )

It builds one choice per output, with a new parser per choice because parsers keep state (vllm/entrypoints/openai/chat_completion/serving.py:980-990). Logprobs are built only when logprobs is true (vllm/entrypoints/openai/chat_completion/serving.py:994-1005). The usage counts the prompt once and the tokens of all choices (vllm/entrypoints/openai/chat_completion/serving.py:1147-1163). With --enable-prompt-tokens-details, it adds the cached prompt tokens (vllm/entrypoints/openai/chat_completion/serving.py:1164-1169).

Step 8: request ids, priority and admission limits#

• Request id. The X-Request-Id header wins over the body field, with or without --enable-request-id-headers; that flag only echoes the id in the response header (Step 2). The engine adds 8 random characters to make the internal id (Data contracts between the parts of vLLM, deep dive "The three request ids").
• Priority. The header X-Vllm-Priority overrides the body field priority, if it parses as an integer (vllm/entrypoints/generate/base/serving.py:42, vllm/entrypoints/generate/base/serving.py:253-265). Only the priority scheduling policy uses it (The V1 scheduler: one token budget, no phases).
• Admission limits. --max-num-queued-reqs and --max-num-queued-tokens are off by default (vllm/config/scheduler.py:92, vllm/config/scheduler.py:105). When set, a full queue gives HTTP 503 with QueueOverflowError or MaxQueuedTokensError, so a load balancer can retry elsewhere (vllm/exceptions.py:162-189). The details are on The engine client: AsyncLLM in the API server.
• DP rank. A router can pin a request to one data-parallel engine with the header X-data-parallel-rank (vllm/entrypoints/generate/base/serving.py:215-228).

Step 9: the other routes#

The full route table is on The OpenAI-compatible API. Some routes have rules that the table does not show:

• GET /health calls check_health() on the engine client: 200, or 503 after EngineDeadError (vllm/entrypoints/serve/instrumentator/health.py:22-33).
• GET /load returns server_load_metrics (vllm/entrypoints/serve/instrumentator/basic.py:30-50). The counter changes only with --enable-server-load-tracking (default off), so without the flag it stays 0 (vllm/entrypoints/serve/utils/api_utils.py:196-197, vllm/entrypoints/launchers/api_server/app_state.py:175-176).
• GET /v1/models lists the base model names and the loaded LoRA adapters (vllm/entrypoints/openai/models/serving.py:149-151).
• POST /tokenize and /detokenize use the same renderer as the chat route (vllm/entrypoints/launchers/api_server/app_state.py:122-130). GET /tokenizer_info exists only with --enable-tokenizer-info-endpoint (vllm/entrypoints/serve/tokenize/api_router.py:86-90).
• /start_profile and /stop_profile exist only when a profiler is configured (vllm/entrypoints/serve/profile/api_router.py:56-65).
• Development routes such as /sleep, /reset_prefix_cache, /collective_rpc and /server_info exist only with VLLM_SERVER_DEV_MODE=1, and the server logs a security warning (vllm/entrypoints/launchers/api_server/routers.py:46-49, vllm/entrypoints/serve/__init__.py:43-65).
• /metrics is the Prometheus endpoint (Metrics and observability: how to watch a vLLM server).

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

curl http://localhost:8000/v1/completions -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-0.5B", "prompt": "The capital of France is",
       "n": 2, "max_tokens": 8, "temperature": 0.8}'

{"id": "cmpl-3f9c...", "object": "text_completion", "model": "Qwen/Qwen2.5-0.5B",
 "choices": [{"index": 0, "text": " Paris. It is the largest city", "finish_reason": "length", ...},
             {"index": 1, "text": " Paris, which is known for its", "finish_reason": "length", ...}],
 "usage": {"prompt_tokens": 5, "completion_tokens": 16, "total_tokens": 21, ...},
 "system_fingerprint": "vllm-<version>-<hash8>"}

What can go wrong#

• 404 "The model ... does not exist". The model field must be a served name (--served-model-name, else the model path) or a LoRA name (vllm/entrypoints/serve/engine/serving.py:75-84).
• 401 on /v1 only. This is the expected behaviour of --api-key; the paths outside the guarded prefixes are open (warning above). OPTIONS requests are never checked (vllm/entrypoints/serve/middleware/authenticate.py:48-54).
• A browser client fails with a CORS error. No origin is allowed by default; set --allowed-origins (vllm/entrypoints/common/cli_args.py:320-321).
• A proxy closes slow streams. Nothing is sent during a long queue or prefill. Set --sse-keep-alive-interval.
• 503 under load with queue limits set. This is the intended behaviour of --max-num-queued-reqs; the token limit counts whole prompts and can reject early (vllm/v1/engine/async_llm.py:337-341).
• The event loop is the bottleneck. Rendering, detokenization and JSON for all requests share one event loop per API server. More API servers split this work (Step 1).
• /load always shows 0. Turn on --enable-server-load-tracking.

Why it matters. Every request of vllm serve crosses these layers twice. Most serving problems (wrong status codes, early 503s, lost streams behind proxies, an overloaded front end) are visible here before they reach the engine.