vLLM Online Client¶

llm_annotator.clients.vllm_online_client ¶

Online vLLM provider: a running OpenAI-compatible vLLM server.

The counterpart that loads the weights in-process instead of talking to a server is vllm_offline_client. Both share VLLMBaseRuntimeOptions, which lives here.

VLLMBaseRuntimeOptions `dataclass` ¶

VLLMBaseRuntimeOptions(
    max_completion_tokens: int | None = None,
    json_schema: dict[str, Any] | None = None,
    temperature: float | None = None,
    top_p: float | None = None,
    top_k: int | None = None,
    repetition_penalty: float | None = None,
    presence_penalty: float | None = None,
    frequency_penalty: float | None = None,
    stop: list[str] | None = None,
    seed: int | None = None,
    n: int | None = None,
    chat_template_kwargs: dict[str, Any] | None = None,
    extra_body: dict[str, Any] | None = None,
)

Bases: ProviderRuntimeOptions

Shared generation options for both vLLM server and offline clients.

Every field here means the same thing to both vLLM clients, so a step can be moved between vllm_online and vllm_offline without its decoding quietly changing. Fields that only one of the two accepts live on that subclass instead.

Attributes:

Name	Type	Description
`temperature`	`float \| None`	Sampling temperature. `None` uses the model default; `0.0` gives greedy, reproducible decoding.
`top_p`	`float \| None`	Nucleus-sampling probability mass. `None` uses the model default.
`top_k`	`int \| None`	Controls the number of top tokens to consider. Set to -1 to consider all tokens.
`repetition_penalty`	`float \| None`	Penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1 encourage the model to use new tokens; values < 1 encourage repetition.
`presence_penalty`	`float \| None`	Penalty applied to tokens already present in the output.
`frequency_penalty`	`float \| None`	Penalty applied proportional to token frequency in the output.
`stop`	`list[str] \| None`	Optional list of strings that halt generation when produced.
`seed`	`int \| None`	Optional fixed random seed for reproducible generation.
`n`	`int \| None`	Number of independent output sequences to generate per request.
`chat_template_kwargs`	`dict[str, Any] \| None`	Additional kwargs forwarded to the chat template. Pass `{"enable_thinking": True}` here to enable thinking mode.
`extra_body`	`dict[str, Any] \| None`	Any other parameter the backend accepts, merged into the request last. This is the escape hatch for everything the fields above do not name, such as `min_p` or `stop_token_ids`.

to_payload ¶

to_payload() -> dict[str, Any]

Build the request payload shared by both vLLM clients.

chat_template_kwargs and extra_body are deliberately excluded: the two clients place them differently, so each subclass adds them.

Returns:

Type	Description
`dict[str, Any]`	A dict of the parameters both vLLM backends spell identically.

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 72–96

VLLMOnlineRuntimeOptions `dataclass` ¶

VLLMOnlineRuntimeOptions(
    max_completion_tokens: int | None = None,
    json_schema: dict[str, Any] | None = None,
    temperature: float | None = None,
    top_p: float | None = None,
    top_k: int | None = None,
    repetition_penalty: float | None = None,
    presence_penalty: float | None = None,
    frequency_penalty: float | None = None,
    stop: list[str] | None = None,
    seed: int | None = None,
    n: int | None = None,
    chat_template_kwargs: dict[str, Any] | None = None,
    extra_body: dict[str, Any] | None = None,
    add_generation_prompt: bool = True,
    chat_template: str | None = None,
    mm_processor_kwargs: dict[str, Any] | None = None,
)

Bases: VLLMBaseRuntimeOptions

Generation options for the vLLM OpenAI-compatible server.

Extends VLLMBaseRuntimeOptions with server-specific parameters from the /v1/chat/completions extra-params API. See https://docs.vllm.ai/en/latest/serving/openai_compatible_server/#api-reference

Attributes:

Name	Type	Description
`add_generation_prompt`	`bool`	If `True`, appends a generation prompt to each message. Defaults to `True`.
`chat_template`	`str \| None`	Optional chat template string. When omitted the model's default template is used.
`mm_processor_kwargs`	`dict[str, Any] \| None`	Arguments forwarded to the model's multi-modal processor (e.g. `{"num_crops": 4}` for Phi-3-Vision).

to_payload ¶

to_payload() -> dict[str, Any]

Build the flat JSON body for vLLM's own endpoints.

Suitable for a request made directly against the server, where every parameter sits at the top level of the body.

Returns:

Type	Description
`dict[str, Any]`	A dict of vLLM server request parameters, including all shared
`dict[str, Any]`	base fields.

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 136–158

split_payload ¶

split_payload() -> tuple[dict[str, Any], dict[str, Any]]

Split the payload into OpenAI-typed kwargs and an extra_body.

The OpenAI SDK validates chat.completions.create against its own signature, so vLLM's extensions must be nested rather than passed as keyword arguments.

Returns:

Type	Description
`dict[str, Any]`	`(kwargs, extra_body)`, where `kwargs` goes to `create()` and
`dict[str, Any]`	`extra_body` is nested under its `extra_body=` parameter.

Examples:

opts = VLLMOnlineRuntimeOptions(temperature=0.0, top_k=20)
standard, extra = opts.split_payload()
standard
# {'temperature': 0.0}
sorted(extra)
# ['add_generation_prompt', 'top_k']

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 160–191

VLLMOnlineClient ¶

VLLMOnlineClient(
    model: str | None = None,
    base_url: str = "http://localhost:8000/v1",
    on_error: OnError = "warn",
)

Bases: OpenAIClient[VLLMOnlineRuntimeOptions]

Client for a running vLLM OpenAI-compatible server.

Initialize the online vLLM client.

Parameters:

Name	Type	Description	Default
`model`	`str \| None`	Model identifier. When omitted, the server is asked which model it serves.	`None`
`base_url`	`str`	Base URL for the vLLM API endpoint.	`'http://localhost:8000/v1'`
`on_error`	`OnError`	Error behavior when generation fails.	`'warn'`

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 199–222

generate ¶

generate(
    *,
    messages: list[dict[str, str]],
    options: VLLMOnlineRuntimeOptions | None = None,
    gen_kwargs: dict[str, Any] | None = None,
) -> Response

Generate a single response from the vLLM server.

Overridden rather than inherited because vLLM's extensions to the chat API (top_k, chat_template_kwargs, ...) are not part of the OpenAI SDK's typed create() signature and have to be nested under extra_body. batch_generate needs no such split: it posts the body itself.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, str]]`	List of message dicts with "role" and "content" keys.	required
`options`	`VLLMOnlineRuntimeOptions \| None`	Optional generation configuration.	`None`
`gen_kwargs`	`dict[str, Any] \| None`	Additional request parameters, merged last so they take precedence over `options`.	`None`

Returns:

Type	Description
`Response`	A Response object containing the generated response.

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 224–275

batch_generate ¶

batch_generate(
    *,
    messages: list[list[dict[str, str]]],
    options: VLLMOnlineRuntimeOptions | None = None,
    gen_kwargs: dict[str, Any] | None = None,
    use_batch_api: bool = False,
    poll_interval: float = 10.0,
) -> list[Response]

Generate responses for a batch of inputs using vLLM's native batch endpoint.

Sends all conversations in a single request to /v1/chat/completions/batch. The OpenAI Batch API is not supported; passing use_batch_api=True raises a ConfigurationError.

No per-sample token counts on this path

That endpoint reports one usage block for the whole batch rather than one per choice, so num_output_tokens is None on every Response it returns, and a step's {prefix}num_tokens column is None with it. Everything else, including reasoning, is per sample as usual. Use generate or the offline provider when the token counts matter.

Parameters:

Name	Type	Description	Default
`messages`	`list[list[dict[str, str]]]`	List of message lists, where each list is a conversation.	required
`options`	`VLLMOnlineRuntimeOptions \| None`	Optional generation configuration.	`None`
`gen_kwargs`	`dict[str, Any] \| None`	Additional provider-specific generation kwargs that are not covered by the standard options. Has precedence over `options`.	`None`
`use_batch_api`	`bool`	Must be `False`. The OpenAI Batch API is not supported by the vLLM server client.	`False`
`poll_interval`	`float`	Accepted for interface compatibility with `OpenAIClient`. Ignored.	`10.0`

Returns:

Type	Description
`list[Response]`	A list of Response objects, one per input conversation,
`list[Response]`	indexed in the same order as input.

Raises:

Type	Description
`ConfigurationError`	If `use_batch_api=True`.
`ProviderError`	If the batch request fails.

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 277–409

vLLM Online Client¶

llm_annotator.clients.vllm_online_client ¶

VLLMBaseRuntimeOptions dataclass ¶

to_payload ¶

VLLMOnlineRuntimeOptions dataclass ¶

to_payload ¶

split_payload ¶

VLLMOnlineClient ¶

generate ¶

batch_generate ¶

VLLMBaseRuntimeOptions `dataclass` ¶

VLLMOnlineRuntimeOptions `dataclass` ¶