Tool and reasoning parsers: from model text to tool_calls and reasoning

Code version: vLLM @ 3e44c0d · 2026-10-22

The idea#

A model writes everything as text. A reasoning model writes its thinking first, for example between <think> and </think>. A model with tools writes a tool call in its own format, for example <tool_call>{"name": ..., "arguments": ...}</tool_call>. The OpenAI API wants three separate fields: reasoning (reasoning output), content and tool_calls (The OpenAI-compatible API).

Two kinds of classes do the split in the API server. A reasoning parser finds where the thinking ends. A tool parser finds tool calls in the rest. Each family of models has its own format, so vLLM has many of each, chosen by name with --reasoning-parser and --tool-call-parser. One Parser object per choice runs both, first the reasoning parser and then the tool parser.

Parsing only reads what the model wrote. To force a tool call (tool_choice "required" or a named function), vLLM also constrains the generation with a grammar (Structured outputs: JSON schemas, grammars and token bitmasks in vLLM). The engine core needs to know when the thinking ends, because the grammar applies only after it. So the reasoning parser runs in the engine core process too.

An everyday comparison. Think of a secretary who sorts a dictated message into three trays: private notes, the letter, and order forms. The secretary knows each writer's habits, for example that one writer always starts notes with "NB". The comparison stops being true in two places. The secretary cannot fix an order form that the writer filled in wrongly; a malformed tool call stays text. And for "required", vLLM does not only sort: it stops the writer from writing anything other than an order form.

The picture#

Read panel 2 from top to bottom; it is the worked example of the first deep dive. Rows that say "None" send no SSE chunk at all. The text of the markers reaches the parsers because the Hermes parser turns off skip_special_tokens for requests with tools (vllm/tool_parsers/hermes_tool_parser.py:59-68).

In the code#

Step 1: the flags#

• --enable-auto-tool-choice plus --tool-call-parser NAME turn on tool parsing. The first without the second stops the start-up with TypeError (vllm/entrypoints/common/cli_args.py:523-525). An unknown parser name with auto tool choice raises KeyError at start-up (vllm/entrypoints/common/cli_args.py:586-594).
• --reasoning-parser NAME turns on reasoning parsing; none means unset (vllm/engine/arg_utils.py:188-190). The flag is a field of StructuredOutputsConfig, because the engine core needs it too (vllm/config/structured_outputs.py:34-39, vllm/engine/arg_utils.py:2187-2204). For gpt-oss models, an empty value becomes openai_gptoss (vllm/model_executor/models/config.py:543-546).
• --tool-parser-plugin and --reasoning-parser-plugin import your own parser module (Plugins in the API server).
• --tool-strict-level auto|function|parameter sets how strictly tool calls are constrained (Step 4).

Step 2: the registries#

ToolParserManager registers 58 built-in names lazily: module path and class name, imported on first use (vllm/tool_parsers/__init__.py:24-270). Examples are hermes, llama3_json, mistral, pythonic, qwen3_coder, deepseek_v3 and openai. ReasoningParserManager registers 42 names the same way, for example deepseek_r1, qwen3, granite, mistral and openai_gptoss (vllm/reasoning/__init__.py:22-198). (Counts of the dict entries at this commit.) The lookup, the lazy import and the plugin flags are on Plugins in the API server.

Step 3: one Parser class for both#

ParserManager.get_parser composes the two named classes into one Parser class. OpenAIServingChat calls it once at start-up (vllm/entrypoints/openai/chat_completion/serving.py:167-175), and so does OnlineRenderer (vllm/renderers/online_renderer.py:180-188). The tool half has a condition:

    @classmethod
    def get_tool_parser(
        cls,
        tool_parser_name: str | None = None,
        enable_auto_tools: bool = False,
        model_name: str | None = None,
    ) -> type[ToolParser] | None:
        """Get the tool parser based on the name."""
        from vllm.tool_parsers import ToolParserManager

        parser: type[ToolParser] | None = None
        if not enable_auto_tools or tool_parser_name is None:
            return parser
        logger.info_once('"auto" tool choice has been enabled.')

        try:
            if (
                tool_parser_name == "pythonic"
                and model_name
                and model_name.startswith("meta-llama/Llama-3.2")
            ):
                logger.warning(
                    "Llama3.2 models may struggle to emit valid pythonic tool calls"
                )
            parser = ToolParserManager.get_tool_parser(tool_parser_name)
        except Exception as e:
            raise TypeError(
                "Error: --enable-auto-tool-choice requires "
                f"tool_parser:'{tool_parser_name}' which has not "
                "been registered"
            ) from e
        return parser

_compose_parser then picks the class (vllm/parser/parser_manager.py:151-274):

1. gpt-oss (Harmony format): always HarmonyParser (lines 179-185).
2. Parsers that read a response template from the tokenizer: one template class. A reasoning parser and a tool parser of two different template kinds cannot be combined (lines 187-220).
3. Three families with their own composite class: kimi_k3, cohere_command3/4 and minicpmv (lines 222-262).
4. Everything else: a subclass of DelegatingParser that holds the reasoning class and the tool class (lines 264-274).

At start-up, validate_parser_tokenizer builds one instance with the model's tokenizer. A parser that cannot find its marker tokens stops the server with TypeError (vllm/parser/parser_manager.py:74-95). For example, a <think>-based reasoning parser raises when the vocabulary has no such token (vllm/reasoning/basic_parsers.py:61-67).

Step 4: the tool_choice modes and how each is enforced#

The request validators set and check tool_choice before anything runs: tools without tool_choice means "auto", and a named function must be one of tools (vllm/entrypoints/openai/chat_completion/protocol.py:975-1042). After rendering, the parser's adjust_request may add a constraint (vllm/renderers/online_renderer.py:824-859, vllm/parser/abstract_parser.py:493-501):

There are two ways to build the constraint for "required" and named calls:

1. Structural tag. Some tool parsers set structural_tag_model, for example hermes, llama3_json, qwen3_coder, kimi_k2 and deepseek_v3. They get a grammar of their own call syntax from xgrammar's templates (vllm/tool_parsers/abstract_tool_parser.py:180-199). _apply_structural_tag writes it into request.structured_outputs and clears response_format (vllm/parser/abstract_parser.py:503-608). This needs VLLM_ENFORCE_STRICT_TOOL_CALLING, which is on by default (vllm/envs.py:1845-1847).
2. JSON schema. Other parsers get a JSON schema: an array of {name, parameters} objects for "required", or the function's parameter schema for a named call (vllm/tool_parsers/utils.py:449-490, vllm/tool_parsers/abstract_tool_parser.py:122-178). The output is then plain JSON, and the serving layer reads it without the model's own markers (vllm/parser/abstract_parser.py:423-451).

With "parallel_tool_calls": false, the grammar stops after the first call, and the serving layer drops any call with an index above 0 (vllm/parser/abstract_parser.py:531-535, vllm/entrypoints/serve/utils/tool_calls_utils.py:24-44).

Step 5: a full response#

Without a stream, the full generator calls parser.parse once per choice, on the whole output text (vllm/entrypoints/openai/chat_completion/serving.py:1007-1019). DelegatingParser.parse splits the reasoning first and looks for tool calls in the rest:

    def parse(
        self,
        model_output: str,
        request: ChatCompletionRequest | ResponsesRequest,
        enable_auto_tools: bool = False,
        model_output_token_ids: Sequence[int] = (),
    ) -> tuple[str | None, str | None, list[FunctionCall] | None]:
        self._initialize_history_tool_call_cnt(request)
        reasoning, content = self.extract_reasoning(model_output, request)
        tool_calls, content = self._extract_tool_calls(
            content=content,
            request=request,
            enable_auto_tools=enable_auto_tools,
        )
        return reasoning, content, tool_calls

The serving class then builds the message with reasoning, content and tool_calls. Each call gets an id, chatcmpl-tool-<uuid> by default (vllm/renderers/chat_utils/tool_call.py:27-32). finish_reason becomes "tool_calls" when the message has calls and the engine stopped normally (vllm/entrypoints/openai/chat_completion/serving.py:1110-1117). A call cut by max_tokens keeps finish_reason: "length", so the client can see that it is incomplete (vllm/entrypoints/serve/utils/tool_calls_utils.py:52-58).

Step 6: streaming deltas#

In a stream, each choice has a ChatStreamProcessor with its own parser (vllm/entrypoints/openai/chat_completion/serving.py:540-555). For each engine output it calls parse_delta (vllm/entrypoints/openai/chat_completion/stream_processor.py:63-72). DelegatingParser.parse_delta works in three parts:

        if not state.prompt_reasoning_checked and prompt_token_ids is not None:
            state.prompt_reasoning_checked = True
            self.set_prompt_token_ids(prompt_token_ids)
            if self._reasoning_parser is None or self.is_reasoning_end(
                prompt_token_ids
            ):
                state.reasoning_ended = True
            else:
                # Reasoning is still open at the end of the prompt; let the
                # reasoning parser adjust its initial parsing state so the
                # first generated tokens are classified correctly.
                self._reasoning_parser.adjust_initial_state_from_prompt(
                    prompt_token_ids
                )

        current_text, current_token_ids = state.advance(delta_text, delta_token_ids)
        delta_message: DeltaMessage | None = None
        reasoning_transitioned = False

        # Reasoning extraction
        if self._in_reasoning_phase(state):
            delta_message = self.extract_reasoning_streaming(
                previous_text=state.previous_text,
                current_text=current_text,
                delta_text=delta_text,
                previous_token_ids=state.previous_token_ids,
                current_token_ids=current_token_ids,
                delta_token_ids=delta_token_ids,
            )
            reasoning_parser = self._reasoning_parser
            if reasoning_parser is not None and reasoning_parser.engine_based_streaming:
                should_transition = (
                    reasoning_parser.has_engine_confirmed_reasoning_end()
                )
            else:
                should_transition = self._is_reasoning_end_streaming(
                    current_token_ids, delta_token_ids
                )
            if should_transition:
                state.reasoning_ended = True
                reasoning_transitioned = True
                current_token_ids = self._extract_content_ids(delta_token_ids)

1. Lines 869-882: on the first call, check the prompt. If the prompt already ended the thinking, skip the reasoning phase. For example, the Hunyuan example template writes an empty <think>\n\n</think> when enable_thinking is false (examples/tool_chat_template_hunyuan_a13b.jinja:110-111).
2. Lines 889-897: in the reasoning phase, the reasoning parser turns the delta into reasoning text.
3. Lines 904-909: the phase ends when the end-of-thinking token id is in the delta; the text after it goes on to the tool phase.

In the tool phase, the tool parser gets the text since the end of the thinking (vllm/parser/abstract_parser.py:938-964). A tool parser's extract_tool_calls_streaming returns None when it has nothing new. Then the serving layer sends no chunk, unless logprobs or token ids must go out (vllm/entrypoints/openai/chat_completion/serving.py:731-741). Tool-call deltas follow the OpenAI stream form. The first delta of a call has index, id, type and the name. Later deltas have only index and more arguments text (vllm/tool_parsers/hermes_tool_parser.py:229-262).

With "include_reasoning": false, reasoning deltas are dropped, and a delta with only reasoning sends nothing (vllm/parser/abstract_parser.py:998-1005). Logprobs and token ids are hidden for the whole stream, because they would leak the hidden thinking (vllm/entrypoints/openai/chat_completion/stream_processor.py:35-37).

Step 7: how the end of the thinking is found#

Most reasoning parsers with a start and an end marker derive from BaseThinkingReasoningParser:

    def is_reasoning_end(self, input_ids: Sequence[int]) -> bool:
        start_token_id = self.start_token_id
        end_token_id = self.end_token_id

        for i in range(len(input_ids) - 1, -1, -1):
            if input_ids[i] == start_token_id:
                return False
            if input_ids[i] == end_token_id:
                return True
        return False

    def is_reasoning_end_streaming(
        self, input_ids: Sequence[int], delta_ids: Iterable[int]
    ) -> bool:
        end_token_id = self.end_token_id
        return end_token_id in delta_ids

    def extract_content_ids(self, input_ids: list[int]) -> list[int]:
        """Extract the content after the end tokens."""
        if self.end_token_id not in islice(input_ids, 0, max(0, len(input_ids) - 1)):
            return []
        else:
            return input_ids[input_ids.index(self.end_token_id) + 1 :]

• is_reasoning_end(ids) reads the ids backwards: an end token before a start token means "ended" (lines 71-80). It is used on the prompt and in the engine core.
• is_reasoning_end_streaming only asks whether the end token is in the new ids (lines 82-86).
• extract_content_ids returns the ids after the end token (lines 88-93).

The reasoning parser also counts the thinking tokens, between start and end markers (vllm/reasoning/basic_parsers.py:178-196). The serving layer reports the count as usage.completion_tokens_details.reasoning_tokens, when a reasoning parser is set (vllm/entrypoints/openai/chat_completion/serving.py:1154-1163). In the response, the thinking is in the vLLM field reasoning. An old client may send reasoning_content in earlier messages; the request validator renames it to reasoning (vllm/entrypoints/openai/chat_completion/protocol.py:590-592).

Step 8: the reasoning parser in the engine core#

A grammar must not constrain the thinking. So the engine core loads the same reasoning parser class for its structured output manager (vllm/v1/structured_output/__init__.py:100-112). The front end helps it twice:

1. It computes reasoning_ended from the rendered prompt and sends it in the EngineCoreRequest (vllm/entrypoints/openai/chat_completion/protocol.py:616-633, vllm/entrypoints/openai/chat_completion/serving.py:242-262, vllm/v1/engine/async_llm.py:516-519).
2. It sends the same chat_template_kwargs as reasoning_parser_kwargs, so the engine core's parser instance reads the output like the front end's one (vllm/v1/structured_output/__init__.py:118-131).

With include_reasoning: false, reasoning_ended is True from the start, so the grammar applies from the first token (vllm/entrypoints/openai/chat_completion/protocol.py:625-626). The page Structured outputs: JSON schemas, grammars and token bitmasks in vLLM shows the engine side.

Step 9: the declarative parser engine (vllm/parser/)#

Newer parsers do not write their own streaming code. A ParserEngineConfig describes a format with terminals (marker strings or token ids), state transitions and the event that text produces in each state (vllm/parser/engine/parser_engine_config.py:50-79). ParserEngine runs one such config for both reasoning and tool calls, streaming and non-streaming (vllm/parser/engine/parser_engine.py:111-123). The folder's guide summarizes the layout: one file per model family, and adapters that expose an engine through the old reasoning-parser and tool-parser interfaces (vllm/parser/AGENTS.md:1-26).

The adapters are what the registries load. For example, the reasoning name qwen3 loads Qwen3ParserReasoningAdapter (vllm/reasoning/__init__.py:168-171, vllm/parser/engine/registered_adapters.py:1-33). Adapters set engine_based_streaming = True (vllm/parser/engine/adapters.py:35-45), and DelegatingParser then feeds them deltas, not the growing full text (vllm/parser/abstract_parser.py:83-105, vllm/parser/abstract_parser.py:152-155). For a user, nothing changes: the flags and names stay the same.

Deep dive#

The sections below give the full detail. Open them one at a time, or use Open all deep dives at the top of the page.

<think>Weather is needed.</think>\n\n<tool_call>{"name": "get_weather", "arguments": {"city": "Paris"}}</tool_call><EOS>

What can go wrong#

• Tool calls arrive as plain text in content. --enable-auto-tool-choice is missing, or the parser does not match the model's format (vllm/parser/parser_manager.py:34-35).
• "tool_choice=... requires --tool-call-parser to be set", but it is set. The tool parser needs --enable-auto-tool-choice as well (warning in Step 3).
• The thinking text is in content. No --reasoning-parser, or the template already opened the thinking in the prompt and the parser does not handle that case.
• A client reads an empty reasoning_content. The response field is reasoning (vllm/entrypoints/openai/chat_completion/protocol.py:79-80); docs/features/reasoning_outputs.md describes the rename (docs/features/reasoning_outputs.md:8-9).
• The server does not start with a reasoning parser. The tokenizer has no matching marker token; check that the parser fits the model (vllm/parser/parser_manager.py:84-95).
• A tool call is cut off. finish_reason: "length" means max_tokens ended the call; the arguments may be invalid JSON.

Why it matters. Agents depend on tool_calls and on a clean content. Most "the model does not call tools" reports are a flag or a parser that does not match the model's format, not a model problem.