vLLM Online Client¶

llm_annotator.clients.vllm_online_client ¶

Online vLLM provider: a running OpenAI-compatible vLLM server.

The counterpart that loads the weights in-process instead of talking to a server is vllm_offline_client. Both share VLLMBaseRuntimeOptions, which lives here.

MAX_CONNECTIONS `module-attribute` ¶

MAX_CONNECTIONS = 8192

How many sockets one client may hold open to its server at once.

The OpenAI SDK caps its HTTP client at 1000 connections (DEFAULT_CONNECTION_LIMITS in openai/_constants.py), and a pooled run can ask for more: a batch_size of 1024 times the default max_concurrent_batches_per_client of 4 is 4096 requests in flight through one client. Above the cap httpx queues the rest, which would keep part of a batch from reaching the server that is supposed to schedule it.

VLLMBaseRuntimeOptions `dataclass` ¶

VLLMBaseRuntimeOptions(
    max_completion_tokens: int | None = None,
    output_schema: dict[str, Any] | None = None,
    temperature: float | None = None,
    top_p: float | None = None,
    top_k: int | None = None,
    repetition_penalty: float | None = None,
    presence_penalty: float | None = None,
    frequency_penalty: float | None = None,
    stop: list[str] | None = None,
    seed: int | None = None,
    chat_template_kwargs: dict[str, Any] | None = None,
    extra_body: dict[str, Any] | None = None,
)

Bases: ProviderRuntimeOptions

Shared generation options for both vLLM server and offline clients.

Every field here means the same thing to both vLLM clients, so a step can be moved between vllm_online and vllm_offline without its decoding quietly changing. Fields that only one of the two accepts live on that subclass instead.

Attributes:

Name	Type	Description
`temperature`	`float \| None`	Sampling temperature. `None` uses the model default; `0.0` gives greedy, reproducible decoding.
`top_p`	`float \| None`	Nucleus-sampling probability mass. `None` uses the model default.
`top_k`	`int \| None`	Controls the number of top tokens to consider. Set to -1 to consider all tokens.
`repetition_penalty`	`float \| None`	Penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1 encourage the model to use new tokens; values < 1 encourage repetition.
`presence_penalty`	`float \| None`	Penalty applied to tokens already present in the output.
`frequency_penalty`	`float \| None`	Penalty applied proportional to token frequency in the output.
`stop`	`list[str] \| None`	Optional list of strings that halt generation when produced.
`seed`	`int \| None`	Optional fixed random seed for reproducible generation.
`chat_template_kwargs`	`dict[str, Any] \| None`	Additional kwargs forwarded to the chat template. Pass `{"enable_thinking": True}` here to enable thinking mode.
`extra_body`	`dict[str, Any] \| None`	Any other parameter the backend accepts, merged into the request last. This is the escape hatch for everything the fields above do not name, such as `min_p` or `stop_token_ids`. It cannot ask for more than one response per sample: the clients read the first one and drop the rest, so an `n` above 1 is rejected.

__post_init__ ¶

__post_init__() -> None

Reject an extra_body that asks for more than one response.

Raises:

Type	Description
`ValueError`	If `extra_body` sets `n` to anything but 1.

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 109–115

to_payload ¶

to_payload() -> dict[str, Any]

Build the request payload shared by both vLLM clients.

chat_template_kwargs and extra_body are deliberately excluded: the two clients place them differently, so each subclass adds them.

Returns:

Type	Description
`dict[str, Any]`	A dict of the parameters both vLLM backends spell identically.

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 117–140

VLLMOnlineRuntimeOptions `dataclass` ¶

VLLMOnlineRuntimeOptions(
    max_completion_tokens: int | None = None,
    output_schema: dict[str, Any] | None = None,
    temperature: float | None = None,
    top_p: float | None = None,
    top_k: int | None = None,
    repetition_penalty: float | None = None,
    presence_penalty: float | None = None,
    frequency_penalty: float | None = None,
    stop: list[str] | None = None,
    seed: int | None = None,
    chat_template_kwargs: dict[str, Any] | None = None,
    extra_body: dict[str, Any] | None = None,
    add_generation_prompt: bool = True,
    chat_template: str | None = None,
    mm_processor_kwargs: dict[str, Any] | None = None,
)

Bases: VLLMBaseRuntimeOptions

Generation options for the vLLM OpenAI-compatible server.

Extends VLLMBaseRuntimeOptions with server-specific parameters from the /v1/chat/completions extra-params API. See https://docs.vllm.ai/en/latest/serving/openai_compatible_server/#api-reference

Attributes:

Name	Type	Description
`add_generation_prompt`	`bool`	If `True`, appends a generation prompt to each message. Defaults to `True`.
`chat_template`	`str \| None`	Optional chat template string. When omitted the model's default template is used.
`mm_processor_kwargs`	`dict[str, Any] \| None`	Arguments forwarded to the model's multi-modal processor (e.g. `{"num_crops": 4}` for Phi-3-Vision).

to_payload ¶

to_payload() -> dict[str, Any]

Build the flat JSON body for vLLM's chat-completions route.

An output_schema becomes a response_format of type json_schema, which is the form vLLM 0.29 reads on /v1/chat/completions: structured_outputs_from_response_format (vllm/entrypoints/generate/base/protocol.py) turns it into the engine's StructuredOutputsParams(json=...), and it overrides a structured_outputs block in the same body. split_payload divides the result over the SDK's typed arguments and extra_body.

Returns:

Type	Description
`dict[str, Any]`	A dict of vLLM server request parameters, including all shared
`dict[str, Any]`	base fields.

Examples:

fmt = VLLMOnlineRuntimeOptions(
    output_schema={"type": "object"}
).to_payload()["response_format"]
fmt["type"], fmt["json_schema"]["name"]
# ('json_schema', 'response')

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 179–223

split_payload ¶

split_payload() -> tuple[dict[str, Any], dict[str, Any]]

Split the payload into OpenAI-typed kwargs and an extra_body.

The OpenAI SDK validates chat.completions.create against its own signature, so vLLM's extensions must be nested rather than passed as keyword arguments.

Returns:

Type	Description
`dict[str, Any]`	`(kwargs, extra_body)`, where `kwargs` goes to `create()` and
`dict[str, Any]`	`extra_body` is nested under its `extra_body=` parameter.

Examples:

opts = VLLMOnlineRuntimeOptions(temperature=0.0, top_k=20)
standard, extra = opts.split_payload()
standard
# {'temperature': 0.0}
sorted(extra)
# ['add_generation_prompt', 'top_k']

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 225–256

VLLMOnlineClient ¶

VLLMOnlineClient(
    model: str | None = None,
    base_url: str = "http://localhost:8000/v1",
    max_workers: int | None = None,
    timeout: float = 3600.0,
    max_retries: int = 2,
    on_error: OnError = "warn",
)

Bases: OpenAIClient[VLLMOnlineRuntimeOptions]

Client for a running vLLM OpenAI-compatible server.

A batch is one /v1/chat/completions request per conversation, sent concurrently. The server batches the requests it holds continuously, so they are scheduled together on the GPU, every request carries its own usage (which fills num_output_tokens per sample), and a conversation that fails is a single error Response rather than a failure of the whole batch.

Initialize the online vLLM client.

Parameters:

Name	Type	Description	Default
`model`	`str \| None`	Model identifier. When omitted, the server is asked which model it serves.	`None`
`base_url`	`str`	Base URL for the vLLM API endpoint.	`'http://localhost:8000/v1'`
`max_workers`	`int \| None`	Maximum number of requests `batch_generate` sends at once. `None` sends the whole batch, which is what lets the server schedule it as one workload. Lower it only to protect a server that is shared with other jobs.	`None`
`timeout`	`float`	Seconds one request may spend reading its answer. It covers the wait in the server's queue as well as the generation itself, so a value below the time a full batch needs turns a healthy run into errors. The default of one hour fits a loaded server that holds thousands of prompts. Making the connection has its own, short limit, so a host that drops packets rather than refusing the connection gives `VLLMQueueAnnotator` the errors it evicts the server on within seconds.	`3600.0`
`max_retries`	`int`	How often the OpenAI SDK retries a request. It retries connection errors, request timeouts and the status codes 408, 409, 429 and 5xx, with an exponential backoff of 0.5 to 8 seconds. Two retries cover a server that restarts without letting a broken pool member stall a batch for long.	`2`
`on_error`	`OnError`	Error behavior when generation fails.	`'warn'`

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 273–322

is_healthy ¶

is_healthy(timeout: float = 5.0) -> bool

Check whether the server answers its /health endpoint.

Parameters:

Name	Type	Description	Default
`timeout`	`float`	Maximum number of seconds to wait for an answer.	`5.0`

Returns:

Type	Description
`bool`	Whether the server answers with status 200.

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 350–359

generate ¶

generate(
    *,
    messages: list[dict[str, str]],
    options: VLLMOnlineRuntimeOptions | None = None,
    gen_kwargs: dict[str, Any] | None = None,
) -> Response

Generate a single response from the vLLM server.

Overridden rather than inherited because vLLM's extensions to the chat API (top_k, chat_template_kwargs, ...) are not part of the OpenAI SDK's typed create() signature and have to be nested under extra_body.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, str]]`	List of message dicts with "role" and "content" keys.	required
`options`	`VLLMOnlineRuntimeOptions \| None`	Optional generation configuration.	`None`
`gen_kwargs`	`dict[str, Any] \| None`	Additional request parameters, merged last so they take precedence over `options`.	`None`

Returns:

Type	Description
`Response`	A Response object containing the generated response. A failed
`Response`	request is an error Response when `on_error` is `"warn"` or
`Response`	`"ignore"`.

Raises:

Type	Description
`ProviderError`	If the request fails and `on_error` is `"raise"`.
`ValueError`	If the request asks for more than one response.

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 361–409

batch_generate ¶

batch_generate(
    *,
    messages: list[list[dict[str, str]]],
    options: VLLMOnlineRuntimeOptions | None = None,
    gen_kwargs: dict[str, Any] | None = None,
) -> list[Response]

Generate one response per conversation, all in flight at once.

Each conversation is its own /v1/chat/completions request. vLLM schedules whatever requests it holds as one continuous batch, so the GPU sees the same workload as a single combined request would give it, while the result stays per sample: response, usage and error belong to one conversation and cannot be mixed up.

Parameters:

Name	Type	Description	Default
`messages`	`list[list[dict[str, str]]]`	List of message lists, where each list is a conversation.	required
`options`	`VLLMOnlineRuntimeOptions \| None`	Optional generation configuration.	`None`
`gen_kwargs`	`dict[str, Any] \| None`	Additional provider-specific generation kwargs that are not covered by the standard options. Has precedence over `options`.	`None`

Returns:

Type	Description
`list[Response]`	A list of Response objects, one per input conversation,
`list[Response]`	indexed in the same order as input.

Raises:

Type	Description
`ProviderError`	If a request fails and `on_error` is `"raise"`.
`ValueError`	If the request asks for more than one response.

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 411–447

server_is_healthy ¶

server_is_healthy(base_url: str, timeout: float) -> bool

Check whether a vLLM server answers its /health endpoint.

Parameters:

Name	Type	Description	Default
`base_url`	`str`	Base URL of the server, with or without the `/v1` suffix.	required
`timeout`	`float`	Maximum number of seconds to wait for an answer.	required

Returns:

Type	Description
`bool`	Whether the server answers with status 200.

View source on GitHub: src/llm_annotator/clients/vllm_online_client.py lines 45–61

vLLM Online Client¶

llm_annotator.clients.vllm_online_client ¶

MAX_CONNECTIONS module-attribute ¶

VLLMBaseRuntimeOptions dataclass ¶

__post_init__ ¶

to_payload ¶

VLLMOnlineRuntimeOptions dataclass ¶

to_payload ¶

split_payload ¶

VLLMOnlineClient ¶

is_healthy ¶

generate ¶

batch_generate ¶

server_is_healthy ¶

MAX_CONNECTIONS `module-attribute` ¶

VLLMBaseRuntimeOptions `dataclass` ¶

VLLMOnlineRuntimeOptions `dataclass` ¶