vLLM Offline Client¶

llm_annotator.clients.vllm_offline_client ¶

vLLM offline provider implementation.

VLLMOfflineRuntimeOptions `dataclass` ¶

VLLMOfflineRuntimeOptions(
    max_completion_tokens: int | None = None,
    output_schema: dict[str, Any] | None = None,
    temperature: float | None = None,
    top_p: float | None = None,
    top_k: int | None = None,
    repetition_penalty: float | None = None,
    presence_penalty: float | None = None,
    frequency_penalty: float | None = None,
    stop: list[str] | None = None,
    seed: int | None = None,
    chat_template_kwargs: dict[str, Any] | None = None,
    extra_body: dict[str, Any] | None = None,
    whitespace_pattern: str | None = "[ ]?",
)

Bases: VLLMBaseRuntimeOptions

Generation options for the vLLM offline client.

Extends VLLMBaseRuntimeOptions, which carries everything both vLLM clients spell the same way (temperature, top_p, top_k, repetition_penalty, presence_penalty, frequency_penalty, stop, seed, chat_template_kwargs and extra_body), with the SamplingParams fields that only in-process inference offers.

Attributes:

Name	Type	Description
`max_completion_tokens`	`int \| None`	Maximum number of output tokens. Inherited from `ProviderRuntimeOptions`. Forwarded to `SamplingParams` as `max_tokens`.
`output_schema`	`dict[str, Any] \| None`	Optional JSON schema dict for structured output via guided decoding. Inherited from `ProviderRuntimeOptions`. When provided, vLLM constrains generation to valid JSON matching the schema.
`whitespace_pattern`	`str \| None`	Regex pattern inserted between JSON tokens during guided decoding. Only used when `output_schema` is set.

to_payload ¶

to_payload() -> dict[str, Any]

Build a SamplingParams-compatible payload dict.

The dict can be passed directly to vllm.SamplingParams(**payload). chat_template_kwargs is intentionally excluded; it must be passed separately to LLM.chat().

Returns:

Type	Description
`dict[str, Any]`	A dict of `SamplingParams`-compatible keyword arguments.

Raises:

Type	Description
`ImportError`	If vLLM is not installed.

View source on GitHub: src/llm_annotator/clients/vllm_offline_client.py lines 53–78

VLLMOfflineClient ¶

VLLMOfflineClient(
    model: str,
    *,
    tensor_parallel_size: int = 1,
    max_num_seqs: int = 256,
    gpu_memory_utilization: float = 0.9,
    enforce_eager: bool = False,
    quantization: str | None = None,
    max_model_len: int | None = None,
    max_num_batched_tokens: int | None = None,
    enable_prefix_caching: bool = True,
    enable_chunked_prefill: bool = True,
    language_model_only: bool = True,
    speculative_config: dict[str, Any] | None = None,
    reasoning_parser: str | None = None,
    extra_vllm_kwargs: dict[str, Any] | None = None,
    on_error: OnError = "warn",
)

Bases: Client[VLLMOfflineRuntimeOptions]

Offline vLLM client that runs inference in-process.

Loads the model into GPU memory on construction and uses vLLM's LLM.chat API for batched generation. Supports structured output via JSON schema guided decoding, automatic prefix caching, and chunked prefill. Use as a context manager to ensure GPU resources are released when done.

batch_generate hands every conversation it is given to one LLM.chat call. How many of them run at the same time is vLLM's own decision, governed by max_num_seqs and max_num_batched_tokens against the KV cache that gpu_memory_utilization sized at start-up.

Parameters:

Name	Type	Description	Default
`model`	`str`	Hugging Face model identifier or local path.	required
`tensor_parallel_size`	`int`	Number of GPUs for tensor parallelism.	`1`
`max_num_seqs`	`int`	Maximum number of sequences processed in parallel.	`256`
`gpu_memory_utilization`	`float`	Target fraction of GPU memory to use.	`0.9`
`enforce_eager`	`bool`	Disable CUDA graphs and run in eager mode.	`False`
`quantization`	`str \| None`	Quantization method (e.g. `"fp8"`, `"awq"`).	`None`
`max_model_len`	`int \| None`	Maximum total sequence length the model supports.	`None`
`max_num_batched_tokens`	`int \| None`	Maximum tokens per forward pass.	`None`
`enable_prefix_caching`	`bool`	Enable automatic KV-cache prefix reuse.	`True`
`enable_chunked_prefill`	`bool`	Process prefills in chunks to bound memory.	`True`
`language_model_only`	`bool`	If True, all non-text modalities are disabled, saving some memory.	`True`
`speculative_config`	`dict[str, Any] \| None`	Optional dict of vLLM speculative decoding config parameters.	`None`
`reasoning_parser`	`str \| None`	Name of the vLLM reasoning parser that splits a thinking model's trace from its answer, e.g. `"qwen3"`.	`None`
`extra_vllm_kwargs`	`dict[str, Any] \| None`	Additional keyword arguments forwarded to `vllm.LLM`. Explicit constructor arguments take precedence over any conflicting keys here.	`None`

Examples:

Basic generation:

client = VLLMOfflineClient(
    model="meta-llama/Llama-3.2-3B-Instruct",
    max_model_len=4096,
)
response = client.generate(
    messages=[{"role": "user", "content": "Hello!"}]
)
client.destroy()

Context manager (recommended):

with VLLMOfflineClient(
    model="meta-llama/Llama-3.2-3B-Instruct",
    max_model_len=4096,
) as client:
    responses = client.batch_generate(
        messages=[
            [{"role": "user", "content": "Hello!"}],
            [{"role": "user", "content": "What is 2+2?"}],
        ]
    )

Structured output with JSON schema:

schema = {
    "type": "object",
    "properties": {"label": {"type": "string"}},
    "required": ["label"],
}
opts = VLLMOfflineRuntimeOptions(
    max_completion_tokens=128, output_schema=schema
)
with VLLMOfflineClient(
    model="meta-llama/Llama-3.2-3B-Instruct"
) as client:
    responses = client.batch_generate(
        messages=[
            [{"role": "user", "content": "Classify: great"}]
        ],
        options=opts,
    )

Initialize the offline vLLM client and load the model into memory.

Parameters:

Name	Type	Description	Default
`model`	`str`	Hugging Face model identifier or local path.	required
`tensor_parallel_size`	`int`	Number of GPUs for tensor parallelism.	`1`
`max_num_seqs`	`int`	Maximum number of sequences processed in parallel.	`256`
`gpu_memory_utilization`	`float`	Target fraction of GPU memory to use.	`0.9`
`enforce_eager`	`bool`	Disable CUDA graphs and run in eager mode.	`False`
`quantization`	`str \| None`	Quantization method (e.g. `"fp8"`, `"awq"`).	`None`
`max_model_len`	`int \| None`	Maximum total sequence length the model supports.	`None`
`max_num_batched_tokens`	`int \| None`	Maximum tokens per forward pass.	`None`
`enable_prefix_caching`	`bool`	Enable automatic KV-cache prefix reuse. Particularly beneficial when many prompts share a common prefix (e.g. a system message), since the shared prefix is only encoded once.	`True`
`enable_chunked_prefill`	`bool`	Process prefills in chunks to reduce peak memory usage and improve scheduling efficiency.	`True`
`language_model_only`	`bool`	If `True`, all non-text modalities are disabled, saving some memory. Defaults to `True`.	`True`
`speculative_config`	`dict[str, Any] \| None`	Optional dict of vLLM speculative decoding config parameters.	`None`
`reasoning_parser`	`str \| None`	Name of the vLLM reasoning parser that splits a thinking model's trace from its answer, e.g. `"qwen3"`. `vllm.LLM` does no such splitting itself, so without it the trace stays inline in the generated text.	`None`
`extra_vllm_kwargs`	`dict[str, Any] \| None`	Additional keyword arguments forwarded to `vllm.LLM`. Explicit constructor arguments take precedence over any conflicting keys here.	`None`
`on_error`	`OnError`	Error behavior when generation fails. Defaults to `"warn"`.	`'warn'`

Raises:

Type	Description
`ImportError`	If vLLM is not installed (raised on first use).

View source on GitHub: src/llm_annotator/clients/vllm_offline_client.py lines 163–231

warm_up ¶

warm_up(
    *,
    system_message: str | None = None,
    prompt_prefix: str | None = None,
    options: VLLMOfflineRuntimeOptions | None = None,
) -> None

Prime the KV-cache with a shared prefix before the main workload.

When many prompts share a common system message or prompt prefix, running a single cheap forward pass first ensures the shared tokens are cached before the first real batch, avoiding a cold-start latency spike on the initial batch.

This is a no-op if neither system_message nor prompt_prefix is provided, or if the model has not been loaded yet.

Parameters:

Name	Type	Description	Default
`system_message`	`str \| None`	Optional system message used in every request.	`None`
`prompt_prefix`	`str \| None`	Optional fixed prefix that starts every user turn.	`None`
`options`	`VLLMOfflineRuntimeOptions \| None`	Optional generation options. Only used to derive a base `SamplingParams`; the token budget is forced to 1 for the warm-up run regardless of the value set here.	`None`

Raises:

Type	Description
`ProviderError`	If the warm-up inference call fails.

View source on GitHub: src/llm_annotator/clients/vllm_offline_client.py lines 277–346

generate ¶

generate(
    *,
    messages: list[dict[str, str]],
    options: VLLMOfflineRuntimeOptions | None = None,
    gen_kwargs: dict[str, Any] | None = None,
) -> Response

Generate a single response for a conversation.

Delegates to batch_generate with a single-item batch.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, str]]`	Conversation as a list of role/content dicts.	required
`options`	`VLLMOfflineRuntimeOptions \| None`	Optional generation configuration. Pass a `VLLMOfflineRuntimeOptions` instance to use vLLM-specific settings.	`None`
`gen_kwargs`	`dict[str, Any] \| None`	Additional provider-specific generation kwargs that are not covered by `options`. Has precedence over `options`.	`None`

Returns:

Type	Description
`Response`	A Response object containing the generated text and metadata. A
`Response`	failed call is an error Response when `on_error` is `"warn"` or
`Response`	`"ignore"`.

Raises:

Type	Description
`ProviderError`	If the vLLM call fails or the stop reason is an error condition, and `on_error` is `"raise"`.

View source on GitHub: src/llm_annotator/clients/vllm_offline_client.py lines 443–475

batch_generate ¶

batch_generate(
    *,
    messages: list[list[dict[str, str]]],
    options: VLLMOfflineRuntimeOptions | None = None,
    gen_kwargs: dict[str, Any] | None = None,
) -> list[Response]

Generate responses for a batch of conversations.

Every conversation goes to one LLM.chat call, which returns once all of them are generated. vLLM decides how many run at the same time. Response order matches input order.

Parameters:

Name	Type	Description	Default
`messages`	`list[list[dict[str, str]]]`	List of conversations, where each conversation is a list of role/content dicts.	required
`options`	`VLLMOfflineRuntimeOptions \| None`	Optional generation configuration. Pass a `VLLMOfflineRuntimeOptions` instance to use vLLM-specific settings such as temperature, top-p, or a JSON schema.	`None`
`gen_kwargs`	`dict[str, Any] \| None`	Additional provider-specific generation kwargs that are not covered by `options`. Has precedence over `options`.	`None`

Returns:

Type	Description
`list[Response]`	A list of Response objects, one per input conversation, in the
`list[Response]`	same order as the input. A failed call gives one error Response per
`list[Response]`	conversation when `on_error` is `"warn"` or `"ignore"`.

Raises:

Type	Description
`ProviderError`	If the model is not loaded or the vLLM call fails, and `on_error` is `"raise"`.
`ValueError`	If the request asks for more than one response.

View source on GitHub: src/llm_annotator/clients/vllm_offline_client.py lines 477–574

destroy ¶

destroy() -> None

Free GPU memory and clean up all vLLM resources.

Every step runs even when an earlier one fails, and a failed step is logged at warning level with its name and the exception. Safe to call multiple times; subsequent calls after the first are no-ops. Also invoked automatically when the client is used as a context manager.

View source on GitHub: src/llm_annotator/clients/vllm_offline_client.py lines 591–634

vLLM Offline Client¶

llm_annotator.clients.vllm_offline_client ¶

VLLMOfflineRuntimeOptions dataclass ¶

to_payload ¶

VLLMOfflineClient ¶

warm_up ¶

generate ¶

batch_generate ¶

destroy ¶

VLLMOfflineRuntimeOptions `dataclass` ¶