Skip to content

OpenAI Client

llm_annotator.clients.openai_client

OpenAI provider implementation.

DEFAULT_TIMEOUT module-attribute 

DEFAULT_TIMEOUT = 600.0

Seconds one request may take, the OpenAI SDK's own default.

DEFAULT_TIMEOUT in openai/_constants.py is httpx.Timeout(timeout=600, connect=5.0).

DEFAULT_MAX_RETRIES module-attribute 

DEFAULT_MAX_RETRIES = 2

How often the OpenAI SDK retries a failed request, its own default (DEFAULT_MAX_RETRIES in openai/_constants.py).

CONNECT_TIMEOUT module-attribute 

CONNECT_TIMEOUT = 5.0

Seconds to wait for the TCP connection of one request, apart from the generation itself.

httpx reads a plain float timeout as all four of its limits, so a long timeout would also let a host that drops packets rather than refusing the connection hold a request for the full duration. This is the OpenAI SDK's own connect timeout (DEFAULT_TIMEOUT.connect in openai/_constants.py).

REASONING_FIELDS module-attribute 

REASONING_FIELDS = ('reasoning', 'reasoning_content')

Names an OpenAI-compatible server may use for a reasoning trace it returns outside the message content. Neither is in the OpenAI schema, so both arrive as extra fields on the message model. vLLM emits reasoning; several hosted OpenAI-compatible APIs emit reasoning_content.

OpenAIRuntimeOptions dataclass 

OpenAIRuntimeOptions(
    max_completion_tokens: int | None = None,
    output_schema: dict[str, Any] | None = None,
    frequency_penalty: float | None = None,
    reasoning_effort: Literal[
        "none", "minimal", "low", "medium", "high", "xhigh"
    ]
    | None = None,
    temperature: float | None = None,
    top_p: float | None = None,
    presence_penalty: float | None = None,
)

Bases: ProviderRuntimeOptions

frequency_penalty class-attribute instance-attribute 

frequency_penalty: float | None = None

Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.

reasoning_effort class-attribute instance-attribute 

reasoning_effort: (
    Literal[
        "none", "minimal", "low", "medium", "high", "xhigh"
    ]
    | None
) = None

Only for supported reasoning models. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.

temperature class-attribute instance-attribute 

temperature: float | None = None

What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. We generally recommend altering this or top_p but not both.

top_p class-attribute instance-attribute 

top_p: float | None = None

An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered.

presence_penalty class-attribute instance-attribute 

presence_penalty: float | None = None

Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.

OpenAIClient 

OpenAIClient(
    model: str,
    max_workers: int | None = None,
    base_url: str | None = None,
    api_key: str | None = None,
    timeout: float = DEFAULT_TIMEOUT,
    max_retries: int = DEFAULT_MAX_RETRIES,
    use_batch_api: bool = False,
    batch_poll_interval: float = 10.0,
    on_error: OnError = "warn",
)

Bases: Client[T_OpenAIOptions]

Client wrapper for OpenAI APIs.

Initialize the OpenAI client.

Parameters:

Name Type Description Default
model str

OpenAI model identifier.

 required
max_workers int | None

Maximum number of concurrent worker threads for batch_generate. Lower this value if you are getting rate limited. If set to None, 1 or lower, multithreading will be disabled.

 None
base_url str | None

Base URL for the OpenAI API endpoint.

 None
api_key str | None

OpenAI API key. If omitted, the SDK will use OPENAI_API_KEY from the environment.

 None
timeout float

Seconds one request may take. The default is the SDK's own (DEFAULT_TIMEOUT in openai/_constants.py).

 DEFAULT_TIMEOUT
max_retries int

How often the SDK retries a request it can retry (connection errors, timeouts, and the status codes 408, 409, 429 and 5xx). The default is the SDK's own (DEFAULT_MAX_RETRIES in openai/_constants.py).

 DEFAULT_MAX_RETRIES
use_batch_api bool

Whether batch_generate submits its requests to the OpenAI Batch API instead of sending them over a thread pool. The Batch API has a completion window of up to 24 hours and costs less, at the price of latency.

 False
batch_poll_interval float

Seconds between two status polls of a running Batch API job. Only read when use_batch_api is True.

 10.0
on_error OnError

Error behavior when generation fails. Valid options are: - "warn": log a warning and return a Response with error set (the default). - "ignore": return that error Response without the warning. - "raise": raise a ProviderError.

 'warn'

View source on GitHub: src/llm_annotator/clients/openai_client.py lines 151–208

destroy 

destroy() -> None

Cancel the batches still tracked and delete the files they own.

A cancellation or deletion that fails is logged at warning level and does not stop the remaining clean-up.

View source on GitHub: src/llm_annotator/clients/openai_client.py lines 274–287

generate 

generate(
    *,
    messages: list[dict[str, str]],
    options: T_OpenAIOptions | None = None,
    gen_kwargs: dict[str, Any] | None = None,
) -> Response

Generate a response using OpenAI.

Parameters:

Name Type Description Default
messages list[dict[str, str]]

List of message dictionaries.

 required
options T_OpenAIOptions | None

Optional generation configuration.

 None
gen_kwargs dict[str, Any] | None

Additional provider-specific generation kwargs that are not covered by the standard options. Has precedence over options.

 None

Returns:

Type Description
Response

A Response object containing the generated response. A failed
Response

request is an error Response when on_error is "warn" or
Response

"ignore".

Raises:

Type Description
ProviderError

If the request fails and on_error is "raise".
ValueError

If the request asks for more than one response.

View source on GitHub: src/llm_annotator/clients/openai_client.py lines 507–559

batch_generate 

batch_generate(
    *,
    messages: list[list[dict[str, str]]],
    options: T_OpenAIOptions | None = None,
    gen_kwargs: dict[str, Any] | None = None,
) -> list[Response]

Generate responses for a batch of inputs.

The requests go out over a thread pool, or as one OpenAI Batch API job when the client was built with use_batch_api=True.

Parameters:

Name Type Description Default
messages list[list[dict[str, str]]]

List of message lists, one per request.

 required
options T_OpenAIOptions | None

Optional generation configuration.

 None
gen_kwargs dict[str, Any] | None

Additional provider-specific generation kwargs that are not covered by the standard options. Has precedence over options.

 None

Returns:

Type Description
list[Response]

A list of Response objects in the same order as the input. A
list[Response]

request that fails is an error Response when on_error is
list[Response]

"warn" or "ignore".

Raises:

Type Description
ProviderError

If a request fails and on_error is "raise".

View source on GitHub: src/llm_annotator/clients/openai_client.py lines 561–599