Pipeline configuration¶

llm_annotator.config ¶

Declarative configuration for the config-driven annotation pipeline.

As much as possible is validated at config validation time but functional elements like preprocess/postprocess/validation functions are not configurable here. If you need such functionality, you need to write your own Python script that uses the library's API directly.

This module holds the models, their validation and the pure factory parts. The live clients and annotators a config describes are built by pool, which is also where the /health polling and the pool watcher live.

StepKind `module-attribute` ¶

StepKind = Literal[
    "vllm_pool", "vllm_online", "vllm_offline", "api"
]

What a step needs in order to run, as reported by --describe-steps.

LOCAL_DATASET_BUILDERS `module-attribute` ¶

LOCAL_DATASET_BUILDERS = frozenset(
    {
        "arrow",
        "audiofolder",
        "csv",
        "imagefolder",
        "json",
        "pandas",
        "parquet",
        "sql",
        "text",
        "videofolder",
        "webdataset",
        "xml",
    }
)

datasets builder names that read files from the local machine.

A dataset.name from this set makes data_dir and data_files local paths, which resolve against the config file's directory like every other path in a config.

DatasetConfig ¶

Bases: _StrictBase

Source dataset for the first step of a pipeline.

Exactly one of name or path is used: name is handed to datasets.load_dataset (a Hub id or a builder name, e.g. "json" to load a local directory of JSON Lines files via data_dir or data_files), path loads a dataset previously written with save_to_disk. data_dir and data_files only apply to name.

Attributes:

Name	Type	Description
`name`	`str \| None`	Hub dataset id or builder name.
`path`	`Path \| None`	Local directory holding a `save_to_disk` dataset.
`config`	`str \| None`	Dataset configuration name.
`split`	`str \| None`	Split to load. Required when the dataset has several splits.
`data_dir`	`str \| None`	Data directory for local/loader datasets.
`data_files`	`str \| list[str] \| dict[str, str \| list[str]] \| None`	Specific file(s) for local/loader datasets, as a single path, a list of paths, or a mapping of split name to path(s).
`max_num_samples`	`int \| None`	Truncate the dataset to this many samples. A re-run with a higher value annotates only the new samples, as long as `shuffle_seed` and the source stay the same.
`shuffle_seed`	`int \| None`	Shuffle the dataset with this seed before truncating.

is_local_source ¶

is_local_source(root: Path) -> bool

Check whether name loads files from this machine.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths resolve against.	required

Returns:

Type	Description
`bool`	`True` for a packaged builder such as `json` or `csv`, and
`bool`	for a `name` that is a directory on disk. `False` for a Hub
`bool`	dataset id, whose `data_files` are patterns inside the
`bool`	repository.

Examples:

from pathlib import Path
DatasetConfig(name="json").is_local_source(Path("."))
# True
DatasetConfig(name="stanfordnlp/imdb").is_local_source(
    Path(".")
)
# False

View source on GitHub: src/llm_annotator/config.py lines 410–435

resolved_data_dir ¶

resolved_data_dir(root: Path) -> str | None

Get data_dir with a relative local path made absolute.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths resolve against.	required

Returns:

Type	Description
`str \| None`	The directory as `load_dataset` should see it, or `None` when
`str \| None`	the block names none.

View source on GitHub: src/llm_annotator/config.py lines 437–449

resolved_data_files ¶

resolved_data_files(
    root: Path,
) -> str | list[str] | dict[str, str | list[str]] | None

Get data_files with relative local paths made absolute.

Every shape the key accepts is kept: a single path, a list, or a mapping of split name to either. Glob patterns are prefixed with root rather than expanded, so datasets still resolves them.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths resolve against.	required

Returns:

Type	Description
`str \| list[str] \| dict[str, str \| list[str]] \| None`	The files as `load_dataset` should see them, or `None` when
`str \| list[str] \| dict[str, str \| list[str]] \| None`	the block names none.

View source on GitHub: src/llm_annotator/config.py lines 451–475

EngineConfig ¶

Bases: _StrictBase

How one vLLM engine is built, for either vLLM provider.

The same field names mean the same thing whether the model is loaded in process or served by vllm serve; only the transport differs. A vllm_offline step turns this block into vllm.LLM keyword arguments, and a vllm_online step whose servers still have to be started turns it into vllm serve flags, which is what llm-annotate --serve-args <step> prints for a job submitter.

That shared spelling is the point: it is why a step states its GPU count once, in tensor_parallel_size, instead of once for the allocation and once for vLLM.

Attributes:

Name	Type	Description
`tensor_parallel_size`	`int`	GPUs one engine shards its weights over. For a served step this is also how many GPUs its job asks for.
`max_model_len`	`int \| None`	Maximum total sequence length, prompt plus completion.
`gpu_memory_utilization`	`float \| None`	Fraction of each GPU's memory vLLM may claim.
`max_num_seqs`	`int \| None`	Sequences the engine runs concurrently.
`max_num_batched_tokens`	`int \| None`	Token budget of a single forward pass.
`enforce_eager`	`bool \| None`	Disable CUDA graphs and run eagerly.
`quantization`	`str \| None`	Quantization method, e.g. `"fp8"` or `"awq"`.
`enable_prefix_caching`	`bool \| None`	Reuse the KV cache of a shared prompt prefix.
`enable_chunked_prefill`	`bool \| None`	Split long prefills to bound peak memory.
`reasoning_parser`	`str \| None`	Name of the vLLM reasoning parser that separates a thinking model's trace from its answer, e.g. `"qwen3"` or `"deepseek_r1"`. Set it and the step writes `{prefix}reasoning` instead of leaving the trace inline in `{prefix}response`. A served step renders it as `--reasoning-parser`; an offline step hands it to the client, which parses the trace itself with vLLM's own parser, since `vllm.LLM` does no such splitting.
`speculative_config`	`dict[str, Any] \| None`	vLLM speculative-decoding configuration.
`extra`	`dict[str, Any]`	Any other vLLM engine argument, by its Python name. Rendered as `--kebab-case` for `vllm serve` and passed verbatim to `vllm.LLM`, so one spelling covers both.

as_llm_kwargs ¶

as_llm_kwargs() -> dict[str, Any]

Build the keyword arguments for an in-process vllm.LLM.

Unset fields are dropped rather than passed as None, so vLLM's own defaults apply to anything the config does not mention. reasoning_parser rides along here because build_client feeds this dict to the offline client's constructor, which keeps it rather than forwarding it: vllm.LLM does not take it.

Returns:

Type	Description
`dict[str, Any]`	Keyword arguments, with `extra` merged in.

Examples:

EngineConfig(max_model_len=4096).as_llm_kwargs()
# {'tensor_parallel_size': 1, 'max_model_len': 4096}

View source on GitHub: src/llm_annotator/config.py lines 529–548

as_serve_args ¶

as_serve_args() -> list[str]

Build the vllm serve flags for one server.

Returned as an argument list rather than a string so a value may contain spaces; --speculative-config takes a JSON object, and shell word-splitting cannot carry one.

Returns:

Type	Description
`list[str]`	Flags for `vllm serve`, without the model or `--host`/`--port`.

Examples:

EngineConfig(max_model_len=4096).as_serve_args()
# ['--tensor-parallel-size', '1', '--max-model-len', '4096']
EngineConfig(enforce_eager=True).as_serve_args()
# ['--tensor-parallel-size', '1', '--enforce-eager']

View source on GitHub: src/llm_annotator/config.py lines 550–578

PoolConfig ¶

Bases: _StrictBase

How many vLLM servers a step wants started for it.

How big each one is lives in EngineConfig instead, because that is a property of the engine rather than of the pool, and a vllm_offline step needs it without wanting a pool at all.

Both keys are reported by llm-annotate --describe-steps, so a job submitter can size the servers it starts and knows how many of them the step needs before it can begin. min_servers is what a pooled run acts on while it runs: it waits for that many servers to answer /health, and the servers that become ready later join the pool. servers is how many the step counts on at full size, which is what the floor on queue_size and the reported concurrency are computed from. A step that talks to servers someone else started (base_urls, hosts_file, url_glob) needs neither key, since base_urls already counts them.

Attributes:

Name	Type	Description
`servers`	`int`	Number of vLLM server processes to run for this step.
`min_servers`	`int`	Number of ready servers required before annotation begins. Defaults to one, so a step starts on the first server that answers and grows as the rest arrive.

ClientConfig ¶

Bases: _StrictBase

Provider, model and execution settings for one step.

init is forwarded verbatim to the client constructor and options to the provider's *RuntimeOptions dataclass, so every provider-specific knob is reachable without this class having to enumerate them.

Supplying base_urls, hosts_file or url_glob (vllm_online only) turns the step into a multi-server run backed by VLLMQueueAnnotator.

Attributes:

Name	Type	Description
`provider`	`ProviderName`	Provider name, spelled exactly `openai`, `claude`, `vllm_online` or `vllm_offline`. No other spelling is accepted, so a typo cannot silently pick a different backend.
`model`	`str \| None`	Model identifier. Optional only for `vllm_online`, which can ask the server which model it serves.
`init`	`dict[str, Any]`	Extra keyword arguments for the client constructor.
`options`	`dict[str, Any]`	Fields of the provider's runtime-options dataclass.
`batch_size`	`int`	Samples per inference batch.
`num_proc`	`int \| None`	Processes used for dataset preprocessing. Use `null` to disable multiprocessing.
`base_urls`	`list[str]`	vLLM server base URLs, for a multi-server pool.
`hosts_file`	`Path \| None`	File with one vLLM base URL per line.
`url_glob`	`str \| None`	Glob matching files that each hold one vLLM base URL. It may be absolute, which is what a job scheduler writing into a scratch directory needs.
`queue_size`	`int \| None`	Batches kept in flight across the pool. `null` derives it from the pool, at four batches per concurrent batch slot. A pool runs `servers` times `max_concurrent_batches_per_client` batches at once, and a queue below that number would leave servers idle, so it is rejected. Sizes a pool, so it is accepted for `vllm_online` only.
`max_concurrent_batches_per_client`	`int`	Maximum number of batches handled at once by each vLLM server. Independent of `batch_size`: each batch carries `batch_size` samples, so one server holds up to their product. Sizes a pool, so it is accepted for `vllm_online` only.
`wait_for_servers`	`float`	Seconds to wait for the `/health` endpoints of `pool.min_servers` servers, or of every server the pool source names when it names fewer, before starting. `0` disables the check.
`engine`	`EngineConfig`	How this step's vLLM engine is built. Applies to both vLLM providers; rejected for the hosted ones.
`pool`	`PoolConfig`	How many servers this step wants. Only meaningful for `vllm_online` steps whose servers are started for them.
`gen_kwargs`	`dict[str, Any]`	Extra request parameters merged over `options` on every generation call, for anything the options dataclass does not name.

configured_servers ¶

configured_servers() -> int

Count the vLLM servers this config accounts for.

pool.servers is what a job submitter is asked to start; explicit base_urls are servers that already exist. A hosts_file or url_glob is not read here, since it may not exist yet when the config is validated, so such a step counts pool.servers alone.

Returns:

Type	Description
`int`	The larger of `pool.servers` and the number of distinct
`int`	`base_urls`.

Examples:

ClientConfig(
    provider="vllm_online", model="m"
).configured_servers()
# 1
ClientConfig(
    provider="vllm_online",
    model="m",
    base_urls=[
        "http://node01:8000/v1",
        "http://node02:8000/v1",
    ],
).configured_servers()
# 2

View source on GitHub: src/llm_annotator/config.py lines 872–899

concurrency ¶

concurrency() -> dict[str, int | None]

Report how many requests and samples this step keeps in flight.

This is what a user needs in order to size a vLLM server's max_num_seqs, and it is reported by llm-annotate --describe-steps before any server job is submitted. The pool keys are None for a provider without a pool, which runs one batch at a time.

Returns:

Type	Description
`dict[str, int \| None]`	A mapping with `batch_size`,
`dict[str, int \| None]`	`max_concurrent_batches_per_client`, the effective
`dict[str, int \| None]`	`queue_size`, `max_requests_per_server`
`dict[str, int \| None]`	(`max_concurrent_batches_per_client` times `batch_size`,
`dict[str, int \| None]`	which is what a server's `max_num_seqs` has to cover) and
`dict[str, int \| None]`	`max_requests_in_flight` (that number times
`dict[str, int \| None]`	`configured_servers`).

Examples:

ClientConfig(provider="claude", model="m").concurrency()
# {'batch_size': 256, 'max_concurrent_batches_per_client': None, 'queue_size': None, 'max_requests_per_server': None, 'max_requests_in_flight': None}
ClientConfig(
    provider="vllm_online",
    model="m",
    batch_size=64,
    pool={"servers": 4},
).concurrency()
# {'batch_size': 64, 'max_concurrent_batches_per_client': 4, 'queue_size': 64, 'max_requests_per_server': 256, 'max_requests_in_flight': 1024}

View source on GitHub: src/llm_annotator/config.py lines 901–953

is_pool ¶

is_pool() -> bool

Whether this client describes a pool of vLLM servers.

Returns:

Type	Description
`bool`	`True` when any of the pool discovery keys is set.

View source on GitHub: src/llm_annotator/config.py lines 955–961

kind ¶

kind() -> StepKind

Classify what a step on this client needs in order to run.

This is the whole taxonomy a job submitter needs, and it is derived rather than configured, so a config cannot disagree with itself about which resources a step wants. A kind names the resources a step needs, not its provider, which is why provider vllm_online can yield either vllm_pool or vllm_online:

vllm_pool Provider vllm_online, but no servers were named, so they still have to be started for it. vllm_online Provider vllm_online pointed at servers that someone else started. vllm_offline It loads the model in-process, so it needs GPUs wherever the annotation itself runs. api A hosted provider; no accelerator at all.

Returns:

Type	Description
`StepKind`	The step kind.

Examples:

ClientConfig(provider="vllm_online", model="m").kind()
# 'vllm_pool'
ClientConfig(
    provider="vllm_online",
    model="m",
    base_urls=["http://node01:8000/v1"],
).kind()
# 'vllm_online'
ClientConfig(provider="claude", model="m").kind()
# 'api'

View source on GitHub: src/llm_annotator/config.py lines 963–1003

cache_key ¶

cache_key() -> str

Build a key identifying the underlying client resources.

Two steps whose keys match can share one live client, which matters because loading a vLLM model takes minutes. Only constructor-level settings appear here: options and gen_kwargs are per-request and are passed to batch_generate, so they never require a rebuild. engine does, because it decides how the engine itself is built.

Returns:

Type	Description
`str`	A stable string key.

View source on GitHub: src/llm_annotator/config.py lines 1005–1031

resolve_base_urls ¶

resolve_base_urls(root: Path) -> list[str]

Collect the pool's base URLs from whichever source was configured.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths and globs resolve against.	required

Returns:

Type	Description
`list[str]`	The base URLs, in file/glob order.

Raises:

Type	Description
`ValueError`	If the configured source yields no URL.

View source on GitHub: src/llm_annotator/config.py lines 1033–1076

build_options ¶

build_options() -> ProviderRuntimeOptions

Instantiate the provider's runtime-options dataclass.

Returns:

Type	Description
`ProviderRuntimeOptions`	The populated options instance.

View source on GitHub: src/llm_annotator/config.py lines 1078–1084

StepConfig ¶

Bases: _StrictBase

One annotation pass over the dataset produced by the previous step.

Prompts and schemas may be given inline or as a file path, never both. File paths resolve against the directory holding the config file, so a config directory can be moved or shared as a unit.

Attributes:

Name	Type	Description
`name`	`str`	Unique step name. Drives the step directory and, by default, the `task_prefix` that namespaces this step's output columns. It holds letters, digits, `_`, `-` and `.`, and starts with a letter or a digit.
`type`	`StepType`	`"annotate"` runs over the incoming dataset; `"generate"` synthesises a dataset from `prompts` and must come first.
`prompt`	`str \| None`	Inline prompt template with `{column}` placeholders.
`prompt_file`	`Path \| None`	File holding the prompt template.
`system_prompt`	`str \| None`	Inline system message.
`system_prompt_file`	`Path \| None`	File holding the system message.
`output_schema`	`dict[str, Any] \| None`	Inline JSON schema for structured output.
`output_schema_file`	`Path \| None`	File holding the JSON schema.
`prompts`	`list[str] \| Path \| None`	Prompts for a `generate` step, or a file with one per line.
`num_samples`	`int \| None`	How often to repeat a single `generate` prompt.
`client`	`dict[str, Any] \| None`	Client overrides merged over the pipeline-level client block. This is a partial block -- it is validated only after merging, by `PipelineConfig.step_client`, so a step can change a single option without repeating `provider` and `model`.
`task_prefix`	`str \| None`	Prefix for this step's internal columns and artifacts. Defaults to `"<name>_"`.
`sort_by_length`	`bool \| Literal['shortest_first', 'longest_first']`	Sort prompts by length for more efficient batching.
`num_retries_invalid`	`int`	Retries for samples that fail schema validation.
`max_samples_per_output_file`	`int \| Literal['auto']`	Samples per JSONL progress file. `"auto"` (the default) is one percent of the rows with a floor of 1000, so at most 100 files are written and a resume stays cheap. A fixed number trades the samples lost at a crash against the cost of rescanning the files on every resume; 0 writes a single file of unlimited size.
`max_consecutive_failed_batches`	`int`	Abort the step once this many batches in a row come back with every sample errored, instead of continuing to burn compute against an unresponsive backend. 0 disables the check.
`upload_every_n_samples`	`int \| None`	Hub progress-backup cadence, 10000 rows by default. Needs `hub_id`; without one nothing is uploaded. Set it to 0 to switch the backup off.
`hub_id`	`str \| None`	Optional Hub dataset id for this step's prepared-data and progress backup, which makes a crashed step resumable from the Hub.
`rename`	`dict[str, str]`	Mapping from produced column name to its final name.
`drop_columns`	`list[str]`	Columns to remove after the step finishes.
`filter_invalid`	`bool`	Drop rows whose schema validation still failed after all retries. Requires an output schema.
`keep_messages`	`bool`	Keep this step's rendered `messages` column instead of dropping it once the step is done.
`force_data_preparation`	`bool`	Rebuild prepared data even if it is cached.

resolved_task_prefix ¶

resolved_task_prefix() -> str

Get the prefix namespacing this step's columns and artifacts.

Returns:

Type	Description
`str`	The explicit `task_prefix` when set, else `"<name>_"`.

View source on GitHub: src/llm_annotator/config.py lines 1234–1242

resolved_prompt ¶

resolved_prompt(root: Path) -> str | None

Get the prompt template, reading prompt_file when needed.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths resolve against.	required

Returns:

Type	Description
`str \| None`	The prompt template, or `None` for a `generate` step that has
`str \| None`	no extra template.

View source on GitHub: src/llm_annotator/config.py lines 1244–1258

resolved_system_prompt ¶

resolved_system_prompt(root: Path) -> str | None

Get the system message, reading system_prompt_file when needed.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths resolve against.	required

Returns:

Type	Description
`str \| None`	The system message, or `None` when the step has none.

View source on GitHub: src/llm_annotator/config.py lines 1260–1273

resolved_output_schema ¶

resolved_output_schema(root: Path) -> dict[str, Any] | None

Get the JSON schema, reading output_schema_file when needed.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths resolve against.	required

Returns:

Type	Description
`dict[str, Any] \| None`	The schema mapping, or `None` when the step has none.

Raises:

Type	Description
`FileNotFoundError`	If the schema file does not exist.
`ValueError`	If the schema file does not decode to a mapping.

View source on GitHub: src/llm_annotator/config.py lines 1275–1300

resolved_prompts ¶

resolved_prompts(root: Path) -> list[str]

Get the prompt list for a generate step.

A .json path is read as a list of strings; any other path is read as a file with one prompt per line, where blank lines are skipped. A single prompt is repeated num_samples times, mirroring generate_dataset.

Parameters:

Name	Type	Description	Default
`root`	`Path`	Directory that relative paths resolve against.	required

Returns:

Type	Description
`list[str]`	The prompts, one per sample to generate.

Raises:

Type	Description
`ValueError`	If no prompt could be resolved, or if a `.json` file does not hold a non-empty list of strings.

View source on GitHub: src/llm_annotator/config.py lines 1302–1338

PipelineConfig ¶

Bases: _StrictBase

A complete, sequentially executed annotation pipeline.

Each step annotates the dataset produced by the step before it, so a later prompt can reference columns an earlier step created. Every step writes its own subdirectory under output_dir and is skipped on a re-run once it has finished, which makes a long pipeline restartable.

Attributes:

Name	Type	Description
`output_dir`	`Path`	Root directory for all step artifacts and the final result. A relative value resolves against `config_dir`, same as every other path in the config.
`steps`	`list[StepConfig]`	The steps to run, in order. At least one is required.
`dataset`	`DatasetConfig \| None`	Input dataset for the first step. Not needed when the first step is a `generate` step.
`client`	`ClientConfig \| None`	Default client settings, merged into every step's own `client`. Optional: a pipeline whose steps each name their own provider and model needs no shared default at all.
`hub_id`	`str \| None`	Optional Hub dataset id for the final dataset. Per-step backups are configured with a step-level `hub_id` instead.
`idx_column`	`str`	Column name used as the stable per-sample identifier that drives resumption. It must not exist in the source dataset. The first step adds it, every step output keeps it, and the final dataset does not have it.
`overwrite`	`bool`	Delete existing step directories before running, discarding any resumable progress.
`verbose`	`bool`	Whether the annotator logs progress information.
`log_level`	`str`	Package log level for the CLI.
`config_dir`	`Path`	Directory that relative paths in this config resolve against. Set automatically by `load_pipeline_config`.

Examples:

config = PipelineConfig(
    output_dir="outputs/demo",
    dataset={"name": "stanfordnlp/imdb", "split": "test"},
    client={"provider": "openai", "model": "gpt-4o-mini"},
    steps=[{"name": "classify", "prompt": "Rate: {text}"}],
)
[step.resolved_task_prefix() for step in config.steps]
# ['classify_']

step_client ¶

step_client(step: StepConfig) -> ClientConfig

Merge a step's client overrides over the pipeline-level defaults.

Either level may be omitted: a pipeline whose steps all share one model needs only the top-level block, and a pipeline whose steps each use a different model needs only the per-step blocks. When both are present, merging is one level deep -- init and options are merged key-by-key so a step can change a single option without repeating the whole block, while other keys are replaced outright.

The one exception is a step that names a different provider. It inherits no key whose value belongs to the provider it was written for: init and options on every switch, plus engine outside the vLLM providers and the pool keys (base_urls, hosts_file, url_glob, pool, queue_size, max_concurrent_batches_per_client, wait_for_servers) outside vllm_online. Dropping init is what keeps one provider's credentials and base URL from reaching another. A key the step writes itself is kept.

The step's block is only validated here, after merging, because on its own it is a fragment that need not name a provider or model. The result is kept per step name, so the merge and its validation run once per step rather than once per caller. Validation fills that cache for every step, and a config is not changed after it is loaded: the command line reaches a step's client through step_client_overrides, which is applied to the decoded mapping before validation.

Parameters:

Name	Type	Description	Default
`step`	`StepConfig`	The step whose effective client settings are wanted.	required

Returns:

Type	Description
`ClientConfig`	The effective client configuration for that step.

Raises:

Type	Description
`ValueError`	If the step ends up with no client at all, or if the merged result is not a valid client configuration. The offending step is named either way.

View source on GitHub: src/llm_annotator/config.py lines 1456–1538

step_dir ¶

step_dir(index: int) -> Path

Get the directory holding one step's artifacts.

Parameters:

Name	Type	Description	Default
`index`	`int`	Zero-based index of the step.	required

Returns:

Type	Description
`Path`	`<output_dir>/<NN>-<name>`, numbered from 1.

View source on GitHub: src/llm_annotator/config.py lines 1540–1550

describe_steps ¶

describe_steps() -> list[dict[str, Any]]

Summarise what each step needs in order to run.

This is the machine-readable half of the config, meant for a job submitter that has to start the right resources for each step without parsing YAML itself. Everything here is derived from the config, so the submitter cannot disagree with the run about which model a step uses.

Returns:

Type	Description
`list[dict[str, Any]]`	One mapping per step, in pipeline order, each with the step's
`list[dict[str, Any]]`	`index` (from 1), `name`, `kind`, `provider`, `model`,
`list[dict[str, Any]]`	the `servers` / `gpus_per_vllm_server` it wants, the
`list[dict[str, Any]]`	`min_servers` it needs before it can start, and the effective
`list[dict[str, Any]]`	concurrency from
`list[dict[str, Any]]`	`ClientConfig.concurrency`.

View source on GitHub: src/llm_annotator/config.py lines 1552–1585

load_config_file ¶

load_config_file(path: str | Path) -> dict[str, Any]

Read a JSON or YAML config file into a plain dictionary.

The format is chosen from the file suffix: .json is parsed as JSON, .yaml and .yml as YAML. YAML is a superset of JSON, so an unknown suffix is parsed as YAML.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Path to the config file.	required

Returns:

Type	Description
`dict[str, Any]`	The decoded mapping.

Raises:

Type	Description
`FileNotFoundError`	If `path` does not exist.
`ValueError`	If the file does not decode to a mapping.

View source on GitHub: src/llm_annotator/config.py lines 78–110

load_pipeline_config ¶

load_pipeline_config(
    path: str | Path,
    overrides: dict[str, Any] | None = None,
    step_client_overrides: dict[str, dict[str, Any]]
    | None = None,
) -> PipelineConfig

Load and validate a pipeline config from a JSON or YAML file.

config_dir is set to the file's parent directory, so every relative path inside the config resolves against the config file rather than the current working directory.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Path to the config file.	required
`overrides`	`dict[str, Any] \| None`	Optional config keys that take precedence over the file. A key may be dotted to reach a nested value, and an integer segment indexes a list, so `{"dataset.max_num_samples": 2000}` and `{"steps.0.client.batch_size": 8}` both set a single value. A missing intermediate block is created.	`None`
`step_client_overrides`	`dict[str, dict[str, Any]] \| None`	Optional per-step client fragments, keyed by step name, merged into that step's own `client` block before validation. This is how a job runner tells one step -- and only that step -- where the servers it should use are, without disturbing steps that run on a different provider.	`None`

Returns:

Type	Description
`PipelineConfig`	The validated pipeline configuration.

Raises:

Type	Description
`ValueError`	If an override key does not fit the shape of the config, or if `step_client_overrides` names a step the config does not define.

View source on GitHub: src/llm_annotator/config.py lines 1588–1640

Pipeline configuration¶

llm_annotator.config ¶

StepKind module-attribute ¶

LOCAL_DATASET_BUILDERS module-attribute ¶

DatasetConfig ¶

is_local_source ¶

resolved_data_dir ¶

resolved_data_files ¶

EngineConfig ¶

as_llm_kwargs ¶

as_serve_args ¶

PoolConfig ¶

ClientConfig ¶

configured_servers ¶

concurrency ¶

is_pool ¶

kind ¶

cache_key ¶

resolve_base_urls ¶

build_options ¶

StepConfig ¶

resolved_task_prefix ¶

resolved_prompt ¶

resolved_system_prompt ¶

resolved_output_schema ¶

resolved_prompts ¶

PipelineConfig ¶

step_client ¶

step_dir ¶

describe_steps ¶

load_config_file ¶

load_pipeline_config ¶

StepKind `module-attribute` ¶

LOCAL_DATASET_BUILDERS `module-attribute` ¶