Base Client¶

llm_annotator.clients.base ¶

Abstract interface for LLM provider clients.

Provider ¶

Bases: StrEnum

Canonical name of the inference backend a client talks to.

Each Client subclass reports one of these through its provider_type. Config files must spell the provider exactly as one of these values; anything else is rejected with a ValueError listing the valid options.

ProviderRuntimeOptions `dataclass` ¶

ProviderRuntimeOptions(
    max_completion_tokens: int | None = None,
    output_schema: dict[str, Any] | None = None,
)

Shared generation options for provider calls; can be subclassed and extended.

Attributes:

Name	Type	Description
`max_completion_tokens`	`int \| None`	Optional maximum number of tokens to generate. This caps the completion only; it does not include the prompt.
`output_schema`	`dict[str, Any] \| None`	Optional JSON schema for structured output. Every client sends it to its provider, which constrains the response to JSON that matches the schema. An `Annotator` sets it from its own `output_schema` argument, which is the only place to give it there.

to_payload ¶

to_payload() -> dict[str, Any]

Convert options to a provider-specific API request payload dict.

Subclasses override this to build the exact kwargs expected by their SDK. The default implementation returns an empty dict.

Returns:

Type	Description
`dict[str, Any]`	A dict of provider-specific request parameters.

View source on GitHub: src/llm_annotator/clients/base.py lines 52–61

Response `dataclass` ¶

Response(
    text: str,
    stop_reason: str | None = None,
    model: str | None = None,
    provider: Provider | None = None,
    num_output_tokens: int | None = None,
    full_response: object | None = None,
    error: str | None = None,
    error_type: str | None = None,
    reasoning: str | None = None,
)

Structured response object returned by provider clients.

reasoning `class-attribute` `instance-attribute` ¶

reasoning: str | None = None

The model's reasoning trace, separated from text.

Set when a thinking model's trace can be told apart from its answer: a vLLM server started with --reasoning-parser, an offline vLLM client given a reasoning_parser name, or a Claude request with a thinking budget. A reasoning model run without a parser returns its trace inside text, tags and all, and this stays None.

Client ¶

Client(
    model: str,
    max_workers: int | None = None,
    on_error: OnError = "warn",
)

Bases: ABC, Generic[T_Options]

Base client interface used by all provider adapters.

Initialize a provider client.

Parameters:

Name	Type	Description	Default
`model`	`str`	Provider-specific model name.	required
`max_workers`	`int \| None`	Maximum number of concurrent worker threads for `batch_generate`. Clients that support native batching may ignore this parameter.	`None`
`on_error`	`OnError`	Error behavior for provider failures. - `"warn"`: log a warning and return a `Response` with `error` set (the default). - `"ignore"`: return that error `Response` without the warning. - `"raise"`: raise a `ProviderError`.	`'warn'`

View source on GitHub: src/llm_annotator/clients/base.py lines 126–154

enter ¶

__enter__() -> Self

Enter the context manager, returning this client instance.

View source on GitHub: src/llm_annotator/clients/base.py lines 221–223

exit ¶

__exit__(exc_type: Any, exc: Any, tb: Any) -> None

Exit the context manager cleanup.

View source on GitHub: src/llm_annotator/clients/base.py lines 225–227

generate `abstractmethod` ¶

generate(
    *,
    messages: list[dict[str, str]],
    options: T_Options | None = None,
    gen_kwargs: dict[str, Any] | None = None,
) -> Response

Generate a response from the provider.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, str]]`	List of message dicts with "role" and "content" keys.	required
`options`	`T_Options \| None`	Provider-specific generation options. NOTE: using this over gen_kwargs is preferred and implemented to facilitate sub-classing and satisfying typing and code-hinting.	`None`
`gen_kwargs`	`dict[str, Any] \| None`	Additional provider-specific generation kwargs that are not covered by the standard options. Has precedence over `options`.	`None`

Returns:

Type	Description
`Response`	A Response object containing the generated response.

View source on GitHub: src/llm_annotator/clients/base.py lines 236–259

batch_generate ¶

batch_generate(
    *,
    messages: list[list[dict[str, str]]],
    options: T_Options | None = None,
    gen_kwargs: dict[str, Any] | None = None,
) -> list[Response]

Generate responses for a batch of inputs.

The default implementation dispatches one generate call per input over a thread pool of max_workers threads. Override this method in subclasses that support native batching (e.g. vLLM offline and vLLM server) for better throughput.

Parameters:

Name	Type	Description	Default
`messages`	`list[list[dict[str, str]]]`	List of message lists, where each message dict has "role" and "content" keys.	required
`options`	`T_Options \| None`	Provider-specific generation options.	`None`
`gen_kwargs`	`dict[str, Any] \| None`	Additional provider-specific generation kwargs that are not covered by the standard options. Has precedence over `options`.	`None`

Returns:

Type	Description
`list[Response]`	One Response per input, in input order. A request that fails is an
`list[Response]`	error Response, unless `on_error` is `"raise"`.

Raises:

Type	Description
`ProviderError`	If a request fails and `on_error` is `"raise"`.

View source on GitHub: src/llm_annotator/clients/base.py lines 318–352

warm_up ¶

warm_up(
    *,
    system_message: str | None = None,
    prompt_prefix: str | None = None,
    options: T_Options | None = None,
) -> None

Prime the client before the main workload (no-op by default).

Override in clients that benefit from a warm-up pass (e.g. VLLMOfflineClient uses this to prime the KV-cache with a shared prefix before the first real batch).

Parameters:

Name	Type	Description	Default
`system_message`	`str \| None`	Optional system message shared across all requests.	`None`
`prompt_prefix`	`str \| None`	Optional fixed prefix that starts every user turn.	`None`
`options`	`T_Options \| None`	Optional generation options used to derive the warm-up params.	`None`

View source on GitHub: src/llm_annotator/clients/base.py lines 354–372

is_healthy ¶

is_healthy(timeout: float = 5.0) -> bool

Check whether the backend can take requests (True by default).

Override in clients whose backend can be probed. VLLMQueueAnnotator calls this after a batch failed entirely, and removes the server from its pool when the answer is False.

Parameters:

Name	Type	Description	Default
`timeout`	`float`	Maximum number of seconds to wait for an answer.	`5.0`

Returns:

Type	Description
`bool`	Whether the backend answers.

View source on GitHub: src/llm_annotator/clients/base.py lines 374–388

destroy ¶

destroy() -> None

Clean up any resources used by the client.

View source on GitHub: src/llm_annotator/clients/base.py lines 390–391

reject_multiple_responses ¶

reject_multiple_responses(payload: dict[str, Any]) -> None

Raise when a request asks the provider for more than one response.

Every client reads the first response of a request and drops the rest, so a higher n spends tokens on output that is never stored. The check covers n at the top level of the payload and inside a nested extra_body, which is where the vLLM server client puts the parameters the OpenAI SDK does not accept.

Parameters:

Name	Type	Description	Default
`payload`	`dict[str, Any]`	The final request payload, after `gen_kwargs` and `extra_body` were merged in.	required

Raises:

Type	Description
`ValueError`	If the payload sets `n` to anything other than 1.

Examples:

reject_multiple_responses({"n": 1, "temperature": 0.0})
reject_multiple_responses(
    {"extra_body": {"n": 4}}
)
# Traceback (most recent call last):
    ...
ValueError: 'n' is 4, but one response per sample is read...

View source on GitHub: src/llm_annotator/clients/base.py lines 64–95

Base Client¶

llm_annotator.clients.base ¶

Provider ¶

ProviderRuntimeOptions dataclass ¶

to_payload ¶

Response dataclass ¶

reasoning class-attribute instance-attribute ¶

Client ¶

__enter__ ¶

__exit__ ¶

generate abstractmethod ¶

batch_generate ¶

warm_up ¶

is_healthy ¶

destroy ¶

reject_multiple_responses ¶

ProviderRuntimeOptions `dataclass` ¶

Response `dataclass` ¶

reasoning `class-attribute` `instance-attribute` ¶

enter ¶

exit ¶

generate `abstractmethod` ¶