Pipeline configuration¶

llm_annotator.config ¶

Declarative configuration for the config-driven annotation pipeline.

As much as possible is validated at config validation time but functional elements like preprocess/postprocess/validation functions are not configurable here. If you need such functionality, you need to write your own Python script that uses the library's API directly.

StepKind `module-attribute` ¶

StepKind = Literal[
    "vllm_pool", "vllm_online", "vllm_offline", "api"
]

What a step needs in order to run, as reported by --describe-steps.

DatasetConfig ¶

Bases: _StrictBase

Source dataset for the first step of a pipeline.

Exactly one of name or path is used: name is handed to datasets.load_dataset (a Hub id or a builder name, e.g. "json" to load a local directory of JSON Lines files via data_dir or data_files), path loads a dataset previously written with save_to_disk. data_dir and data_files only apply to name.

Attributes:

Name	Type	Description
`name`	`str \| None`	Hub dataset id or builder name.
`path`	`Path \| None`	Local directory holding a `save_to_disk` dataset.
`config`	`str \| None`	Dataset configuration name.
`split`	`str \| None`	Split to load. Required when the dataset has several splits.
`data_dir`	`str \| None`	Data directory for local/loader datasets.
`data_files`	`str \| list[str] \| dict[str, str \| list[str]] \| None`	Specific file(s) for local/loader datasets, as a single path, a list of paths, or a mapping of split name to path(s).
`max_num_samples`	`int \| None`	Truncate the dataset to this many samples.
`shuffle_seed`	`int \| None`	Shuffle the dataset with this seed before truncating.

EngineConfig ¶

Bases: _StrictBase

How one vLLM engine is built, for either vLLM provider.

The same field names mean the same thing whether the model is loaded in process or served by vllm serve; only the transport differs. A vllm_offline step turns this block into vllm.LLM keyword arguments, and a vllm_online step whose servers still have to be started turns it into vllm serve flags, which is what llm-annotate --serve-args <step> prints for a job submitter.

That shared spelling is the point: it is why a step states its GPU count once, in tensor_parallel_size, instead of once for the allocation and once for vLLM.

Attributes:

Name	Type	Description
`tensor_parallel_size`	`int`	GPUs one engine shards its weights over. For a served step this is also how many GPUs its job asks for.
`max_model_len`	`int \| None`	Maximum total sequence length, prompt plus completion.
`gpu_memory_utilization`	`float \| None`	Fraction of each GPU's memory vLLM may claim.
`max_num_seqs`	`int \| None`	Sequences the engine runs concurrently.
`max_num_batched_tokens`	`int \| None`	Token budget of a single forward pass.
`enforce_eager`	`bool \| None`	Disable CUDA graphs and run eagerly.
`quantization`	`str \| None`	Quantization method, e.g. `"fp8"` or `"awq"`.
`enable_prefix_caching`	`bool \| None`	Reuse the KV cache of a shared prompt prefix.
`enable_chunked_prefill`	`bool \| None`	Split long prefills to bound peak memory.
`reasoning_parser`	`str \| None`	Name of the vLLM reasoning parser that separates a thinking model's trace from its answer, e.g. `"qwen3"` or `"deepseek_r1"`. Set it and the step writes `{prefix}reasoning` instead of leaving the trace inline in `{prefix}response`. A served step renders it as `--reasoning-parser`; an offline step hands it to the client, which parses the trace itself with vLLM's own parser, since `vllm.LLM` does no such splitting.
`speculative_config`	`dict[str, Any] \| None`	vLLM speculative-decoding configuration.
`extra`	`dict[str, Any]`	Any other vLLM engine argument, by its Python name. Rendered as `--kebab-case` for `vllm serve` and passed verbatim to `vllm.LLM`, so one spelling covers both.

as_llm_kwargs ¶

as_llm_kwargs() -> dict[str, Any]

Build the keyword arguments for an in-process vllm.LLM.

Unset fields are dropped rather than passed as None, so vLLM's own defaults apply to anything the config does not mention. reasoning_parser rides along here because build_client feeds this dict to the offline client's constructor, which keeps it rather than forwarding it: vllm.LLM does not take it.

Returns:

Type	Description
`dict[str, Any]`	Keyword arguments, with `extra` merged in.

Examples:

EngineConfig(max_model_len=4096).as_llm_kwargs()
# {'tensor_parallel_size': 1, 'max_model_len': 4096}

View source on GitHub: src/llm_annotator/config.py lines 303–322

as_serve_args ¶

as_serve_args() -> list[str]

Build the vllm serve flags for one server.

Returned as an argument list rather than a string so a value may contain spaces; --speculative-config takes a JSON object, and shell word-splitting cannot carry one.

Returns:

Type	Description
`list[str]`	Flags for `vllm serve`, without the model or `--host`/`--port`.

Examples:

EngineConfig(max_model_len=4096).as_serve_args()
# ['--tensor-parallel-size', '1', '--max-model-len', '4096']
EngineConfig(enforce_eager=True).as_serve_args()
# ['--tensor-parallel-size', '1', '--enforce-eager']

View source on GitHub: src/llm_annotator/config.py lines 324–352

PoolConfig ¶

Bases: _StrictBase

How many vLLM servers a step wants started for it.

How big each one is lives in EngineConfig instead, because that is a property of the engine rather than of the pool, and a vllm_offline step needs it without wanting a pool at all.

The library itself never acts on this block; it is reported by llm-annotate --describe-steps so a job submitter can size the servers it starts. A step that talks to servers someone else started (base_urls, hosts_file, url_glob) does not need it.

Attributes:

Name	Type	Description
`servers`	`int`	Number of vLLM server processes to run for this step.

ClientConfig ¶

Bases: _StrictBase

Provider, model and execution settings for one step.

init is forwarded verbatim to the client constructor and options to the provider's *RuntimeOptions dataclass, so every provider-specific knob is reachable without this class having to enumerate them.

Supplying base_urls, hosts_file or url_glob (vllm_online only) turns the step into a multi-server run backed by VLLMQueueAnnotator.

Attributes:

Name	Type	Description
`provider`	`ProviderName`	Provider name, spelled exactly `openai`, `claude`, `vllm_online` or `vllm_offline`. No other spelling is accepted, so a typo cannot silently pick a different backend.
`model`	`str \| None`	Model identifier. Optional only for `vllm_online`, which can ask the server which model it serves.
`init`	`dict[str, Any]`	Extra keyword arguments for the client constructor.
`options`	`dict[str, Any]`	Fields of the provider's runtime-options dataclass.
`batch_size`	`int`	Samples per inference batch.
`num_proc`	`int \| None`	Processes used for dataset preprocessing. Use `null` to disable multiprocessing.
`base_urls`	`list[str]`	vLLM server base URLs, for a multi-server pool.
`hosts_file`	`Path \| None`	File with one vLLM base URL per line.
`url_glob`	`str \| None`	Glob matching files that each hold one vLLM base URL. It may be absolute, which is what a job scheduler writing into a scratch directory needs.
`queue_size`	`int \| None`	Batches kept in flight across the pool.
`wait_for_servers`	`float`	Seconds to wait for every server's `/health` before starting. `0` disables the check.
`engine`	`EngineConfig`	How this step's vLLM engine is built. Applies to both vLLM providers; rejected for the hosted ones.
`pool`	`PoolConfig`	How many servers this step wants. Only meaningful for `vllm_online` steps whose servers are started for them.
`gen_kwargs`	`dict[str, Any]`	Extra request parameters merged over `options` on every generation call, for anything the options dataclass does not name.

is_pool ¶

is_pool() -> bool

Whether this client describes a pool of vLLM servers.

Returns:

Type	Description
`bool`	`True` when any of the pool discovery keys is set.

View source on GitHub: src/llm_annotator/config.py lines 511–517

kind ¶

kind() -> StepKind

Classify what a step on this client needs in order to run.

This is the whole taxonomy a job submitter needs, and it is derived rather than configured, so a config cannot disagree with itself about which resources a step wants. A kind names the resources a step needs, not its provider, which is why provider vllm_online can yield either vllm_pool or vllm_online:

vllm_pool Provider vllm_online, but no servers were named, so they still have to be started for it. vllm_online Provider vllm_online pointed at servers that someone else started. vllm_offline It loads the model in-process, so it needs GPUs wherever the annotation itself runs. api A hosted provider; no accelerator at all.

Returns:

Type	Description
`StepKind`	The step kind.

Examples:

ClientConfig(provider="vllm_online", model="m").kind()
# 'vllm_pool'
ClientConfig(
    provider="vllm_online",
    model="m",
    base_urls=["http://node01:8000/v1"],
).kind()
# 'vllm_online'
ClientConfig(provider="claude", model="m").kind()
# 'api'

View source on GitHub: src/llm_annotator/config.py lines 519–559

cache_key ¶

cache_key() -> str

Build a key identifying the underlying client resources.

Two steps whose keys match can share one live client, which matters because loading a vLLM model takes minutes. Only constructor-level settings appear here: options and gen_kwargs are per-request and are passed to batch_generate, so they never require a rebuild. engine does, because it decides how the engine itself is built.

Returns:

Type	Description
`str`	A stable string key.

View source on GitHub: src/llm_annotator/config.py lines 561–587

resolve_base_urls ¶

resolve_base_urls(root: Path) -> list[str]

Collect the pool's base URLs from whichever source was configured.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths and globs resolve against.	required

Returns:

Type	Description
`list[str]`	The base URLs, in file/glob order.

Raises:

Type	Description
`ValueError`	If the configured source yields no URL.

View source on GitHub: src/llm_annotator/config.py lines 589–632

build_options ¶

build_options(
    output_schema: dict[str, Any] | None = None,
) -> ProviderRuntimeOptions

Instantiate the provider's runtime-options dataclass.

Parameters:

Name	Type	Description	Default
`output_schema`	`dict[str, Any] \| None`	Optional JSON schema for structured output. It is passed through to the annotator rather than set here, so this argument only guards against setting it twice.	`None`

Returns:

Type	Description
`ProviderRuntimeOptions`	The populated options instance.

Raises:

Type	Description
`ValueError`	If `json_schema` is set in `options` while an `output_schema` is also configured for the step.

View source on GitHub: src/llm_annotator/config.py lines 634–656

build_client ¶

build_client(root: Path) -> Client[Any] | list[Client[Any]]

Instantiate the client, or one client per server for a pool.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths and globs resolve against.	required

Returns:

Type	Description
`Client[Any] \| list[Client[Any]]`	A single client, or a list of clients when a pool is configured.

View source on GitHub: src/llm_annotator/config.py lines 658–700

build_annotator ¶

build_annotator(
    root: Path, verbose: bool = False
) -> Annotator

Instantiate the annotator that drives this client.

A pool of servers yields a VLLMQueueAnnotator; everything else yields a plain Annotator.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths and globs resolve against.	required
`verbose`	`bool`	Whether the annotator should log progress information.	`False`

Returns:

Type	Description
`Annotator`	The annotator, ready to run.

View source on GitHub: src/llm_annotator/config.py lines 702–734

StepConfig ¶

Bases: _StrictBase

One annotation pass over the dataset produced by the previous step.

Prompts and schemas may be given inline or as a file path, never both. File paths resolve against the directory holding the config file, so a config directory can be moved or shared as a unit.

Attributes:

Name	Type	Description
`name`	`str`	Unique step name. Drives the step directory and, by default, the `task_prefix` that namespaces this step's output columns.
`type`	`StepType`	`"annotate"` runs over the incoming dataset; `"generate"` synthesises a dataset from `prompts` and must come first.
`prompt`	`str \| None`	Inline prompt template with `{column}` placeholders.
`prompt_file`	`Path \| None`	File holding the prompt template.
`system_prompt`	`str \| None`	Inline system message.
`system_prompt_file`	`Path \| None`	File holding the system message.
`output_schema`	`dict[str, Any] \| None`	Inline JSON schema for structured output.
`output_schema_file`	`Path \| None`	File holding the JSON schema.
`prompts`	`list[str] \| Path \| None`	Prompts for a `generate` step, or a file with one per line.
`num_samples`	`int \| None`	How often to repeat a single `generate` prompt.
`client`	`dict[str, Any] \| None`	Client overrides merged over the pipeline-level client block. This is a partial block -- it is validated only after merging, by `PipelineConfig.step_client`, so a step can change a single option without repeating `provider` and `model`.
`task_prefix`	`str \| None`	Prefix for this step's internal columns and artifacts. Defaults to `"<name>_"`.
`sort_by_length`	`bool \| Literal['shortest_first', 'longest_first']`	Sort prompts by length for more efficient batching.
`num_retries_invalid`	`int`	Retries for samples that fail schema validation.
`max_samples_per_output_file`	`int`	Samples per JSONL progress file.
`max_consecutive_failed_batches`	`int`	Abort the step once this many batches in a row come back with every sample errored, instead of continuing to burn compute against an unresponsive backend. 0 disables the check.
`upload_every_n_samples`	`int \| None`	Hub progress-backup cadence. Needs `hub_id`.
`hub_id`	`str \| None`	Optional Hub dataset id for this step's prepared-data and progress backup, which makes a crashed step resumable from the Hub.
`rename`	`dict[str, str]`	Mapping from produced column name to its final name.
`drop_columns`	`list[str]`	Columns to remove after the step finishes.
`filter_invalid`	`bool`	Drop rows whose schema validation still failed after all retries. Requires an output schema.
`keep_messages`	`bool`	Keep this step's rendered `messages` column instead of dropping it once the step is done.
`force_data_preparation`	`bool`	Rebuild prepared data even if it is cached.

resolved_task_prefix ¶

resolved_task_prefix() -> str

Get the prefix namespacing this step's columns and artifacts.

Returns:

Type	Description
`str`	The explicit `task_prefix` when set, else `"<name>_"`.

View source on GitHub: src/llm_annotator/config.py lines 857–865

resolved_prompt ¶

resolved_prompt(root: Path) -> str | None

Get the prompt template, reading prompt_file when needed.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths resolve against.	required

Returns:

Type	Description
`str \| None`	The prompt template, or `None` for a `generate` step that has
`str \| None`	no extra template.

View source on GitHub: src/llm_annotator/config.py lines 867–881

resolved_system_prompt ¶

resolved_system_prompt(root: Path) -> str | None

Get the system message, reading system_prompt_file when needed.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths resolve against.	required

Returns:

Type	Description
`str \| None`	The system message, or `None` when the step has none.

View source on GitHub: src/llm_annotator/config.py lines 883–896

resolved_output_schema ¶

resolved_output_schema(root: Path) -> dict[str, Any] | None

Get the JSON schema, reading output_schema_file when needed.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths resolve against.	required

Returns:

Type	Description
`dict[str, Any] \| None`	The schema mapping, or `None` when the step has none.

Raises:

Type	Description
`ValueError`	If the schema file does not decode to a mapping.

View source on GitHub: src/llm_annotator/config.py lines 898–922

resolved_prompts ¶

resolved_prompts(root: Path) -> list[str]

Get the prompt list for a generate step.

A path is read as a file with one prompt per line; blank lines are skipped. A single prompt is repeated num_samples times, mirroring generate_dataset.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths resolve against.	required

Returns:

Type	Description
`list[str]`	The prompts, one per sample to generate.

Raises:

Type	Description
`ValueError`	If no prompt could be resolved.

View source on GitHub: src/llm_annotator/config.py lines 924–955

PipelineConfig ¶

Bases: _StrictBase

A complete, sequentially executed annotation pipeline.

Each step annotates the dataset produced by the step before it, so a later prompt can reference columns an earlier step created. Every step writes its own subdirectory under output_dir and is skipped on a re-run once it has finished, which makes a long pipeline restartable.

Attributes:

Name	Type	Description
`output_dir`	`Path`	Root directory for all step artifacts and the final result. A relative value resolves against `config_dir`, same as every other path in the config.
`steps`	`list[StepConfig]`	The steps to run, in order. At least one is required.
`dataset`	`DatasetConfig \| None`	Input dataset for the first step. Not needed when the first step is a `generate` step.
`client`	`ClientConfig \| None`	Default client settings, merged into every step's own `client`. Optional: a pipeline whose steps each name their own provider and model needs no shared default at all.
`hub_id`	`str \| None`	Optional Hub dataset id for the final dataset. Per-step backups are configured with a step-level `hub_id` instead.
`idx_column`	`str`	Column name used as the stable per-sample identifier that drives resumption. It must not exist in the source dataset.
`overwrite`	`bool`	Delete existing step directories before running, discarding any resumable progress.
`verbose`	`bool`	Whether the annotator logs progress information.
`log_level`	`str`	Package log level for the CLI.
`config_dir`	`Path`	Directory that relative paths in this config resolve against. Set automatically by `load_pipeline_config`.

Examples:

config = PipelineConfig(
    output_dir="outputs/demo",
    dataset={"name": "stanfordnlp/imdb", "split": "test"},
    client={"provider": "openai", "model": "gpt-4o-mini"},
    steps=[{"name": "classify", "prompt": "Rate: {text}"}],
)
[step.resolved_task_prefix() for step in config.steps]
# ['classify_']

step_client ¶

step_client(step: StepConfig) -> ClientConfig

Merge a step's client overrides over the pipeline-level defaults.

Either level may be omitted: a pipeline whose steps all share one model needs only the top-level block, and a pipeline whose steps each use a different model needs only the per-step blocks. When both are present, merging is one level deep -- init and options are merged key-by-key so a step can change a single option without repeating the whole block, while other keys are replaced outright.

The one exception is a step that names a different provider: it inherits no options at all, because they name fields of the previous provider's runtime-options dataclass and would be rejected as unknown. Its own options are kept as written.

The step's block is only validated here, after merging, because on its own it is a fragment that need not name a provider or model.

Parameters:

Name	Type	Description	Default
`step`	`StepConfig`	The step whose effective client settings are wanted.	required

Returns:

Type	Description
`ClientConfig`	The effective client configuration for that step.

Raises:

Type	Description
`ValueError`	If the step ends up with no client at all, or if the merged result is not a valid client configuration. The offending step is named either way.

View source on GitHub: src/llm_annotator/config.py lines 1059–1123

step_dir ¶

step_dir(index: int) -> Path

Get the directory holding one step's artifacts.

Parameters:

Name	Type	Description	Default
`index`	`int`	Zero-based index of the step.	required

Returns:

Type	Description
`Path`	`<output_dir>/<NN>-<name>`, numbered from 1.

View source on GitHub: src/llm_annotator/config.py lines 1125–1135

describe_steps ¶

describe_steps() -> list[dict[str, Any]]

Summarise what each step needs in order to run.

This is the machine-readable half of the config, meant for a job submitter that has to start the right resources for each step without parsing YAML itself. Everything here is derived from the config, so the submitter cannot disagree with the run about which model a step uses.

Returns:

Type	Description
`list[dict[str, Any]]`	One mapping per step, in pipeline order, each with the step's
`list[dict[str, Any]]`	`index` (from 1), `name`, `kind`, `provider`, `model` and
`list[dict[str, Any]]`	the `servers` / `gpus_per_vllm_server` it wants.

View source on GitHub: src/llm_annotator/config.py lines 1137–1165

load_config_file ¶

load_config_file(path: str | Path) -> dict[str, Any]

Read a JSON or YAML config file into a plain dictionary.

The format is chosen from the file suffix: .json is parsed as JSON, .yaml and .yml as YAML. YAML is a superset of JSON, so an unknown suffix is parsed as YAML.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Path to the config file.	required

Returns:

Type	Description
`dict[str, Any]`	The decoded mapping.

Raises:

Type	Description
`FileNotFoundError`	If `path` does not exist.
`ValueError`	If the file does not decode to a mapping.

View source on GitHub: src/llm_annotator/config.py lines 47–79

wait_for_servers ¶

wait_for_servers(
    base_urls: list[str], timeout: float
) -> None

Block until every vLLM server answers its /health endpoint.

Parameters:

Name	Type	Description	Default
`base_urls`	`list[str]`	vLLM base URLs (each ending in `/v1`).	required
`timeout`	`float`	Maximum number of seconds to wait per server.	required

Raises:

Type	Description
`TimeoutError`	If a server is still unreachable after `timeout`.

View source on GitHub: src/llm_annotator/config.py lines 170–197

load_pipeline_config ¶

load_pipeline_config(
    path: str | Path,
    overrides: dict[str, Any] | None = None,
    step_client_overrides: dict[str, dict[str, Any]]
    | None = None,
) -> PipelineConfig

Load and validate a pipeline config from a JSON or YAML file.

config_dir is set to the file's parent directory, so every relative path inside the config resolves against the config file rather than the current working directory.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Path to the config file.	required
`overrides`	`dict[str, Any] \| None`	Optional top-level keys that take precedence over the file.	`None`
`step_client_overrides`	`dict[str, dict[str, Any]] \| None`	Optional per-step client fragments, keyed by step name, merged into that step's own `client` block before validation. This is how a job runner tells one step -- and only that step -- where the servers it should use are, without disturbing steps that run on a different provider.	`None`

Returns:

Type	Description
`PipelineConfig`	The validated pipeline configuration.

Raises:

Type	Description
`ValueError`	If `step_client_overrides` names a step the config does not define.

View source on GitHub: src/llm_annotator/config.py lines 1168–1215

Pipeline configuration¶

llm_annotator.config ¶

StepKind module-attribute ¶

DatasetConfig ¶

EngineConfig ¶

as_llm_kwargs ¶

as_serve_args ¶

PoolConfig ¶

ClientConfig ¶

is_pool ¶

kind ¶

cache_key ¶

resolve_base_urls ¶

build_options ¶

build_client ¶

build_annotator ¶

StepConfig ¶

resolved_task_prefix ¶

resolved_prompt ¶

resolved_system_prompt ¶

resolved_output_schema ¶

resolved_prompts ¶

PipelineConfig ¶

step_client ¶

step_dir ¶

describe_steps ¶

load_config_file ¶

wait_for_servers ¶

load_pipeline_config ¶

StepKind `module-attribute` ¶