Pipeline runner¶

llm_annotator.pipeline ¶

Sequential executor and CLI for config-driven annotation pipelines.

run_pipeline walks the steps of a PipelineConfig in order, handing the dataset each step produces to the next one. Nothing here re-implements annotation: a step is one ordinary annotate_dataset call, so prompt templating, JSONL progress checkpoints, resumption, retries and Hub backups behave exactly as they do when the library is used directly.

Two things make a pipeline more than a loop:

Every step owns a subdirectory of output_dir and a task_prefix, so its internal columns and artifacts cannot collide with another step's.
A finished step writes its result to <step-dir>/output. On a re-run that snapshot is loaded and the step is skipped, so a pipeline that dies in step three does not repeat steps one and two.
The idx_column of the first step is the id of a row in every later step. A re-run with a higher dataset.max_num_samples resumes every step and sends only the new rows to the model.

main is the llm-annotate console entry point.

STEP_OUTPUT_SUBDIR `module-attribute` ¶

STEP_OUTPUT_SUBDIR = 'output'

Name of the subdirectory holding a finished step's dataset.

Its presence marks a step as done for the selection in the step's SelectionRecord.

STEP_ANNOTATE_SUBDIR `module-attribute` ¶

STEP_ANNOTATE_SUBDIR = 'annotate'

Name of the subdirectory handed to the annotator as its output_dir.

It is one level below the step directory because _post_annotate writes a dataset into the root of whatever output_dir it is given.

run_pipeline ¶

run_pipeline(
    config: PipelineConfig,
    selected: Sequence[str] | None = None,
    retry_errors: bool | Sequence[str] = False,
) -> Dataset

Run a pipeline, or part of one, and return the resulting dataset.

Steps share one live client whenever their provider, model and constructor settings match, so a pipeline that uses the same local model twice loads it only once. The client is always released before returning, including on failure.

Passing selected runs only those steps. Earlier steps must already have finished: their saved output is loaded as the input, which is what lets a scheduler run one step per job while keeping a single config file as the source of truth.

Parameters:

Name	Type	Description	Default
`config`	`PipelineConfig`	The validated pipeline configuration.	required
`selected`	`Sequence[str] \| None`	Names of the steps to run. `None` runs all of them. The names must form a contiguous run of the pipeline.	`None`
`retry_errors`	`bool \| Sequence[str]`	Annotate rows again that finished with an error. `True` redoes every errored row of the selected steps, a sequence redoes only those error types, e.g. `["ConnectError"]`. A row that is redone in one step is also redone in every selected step after it, because those steps read what it produces.	`False`

Returns:

Type	Description
`Dataset`	The dataset produced by the last step that ran.

Raises:

Type	Description
`ValueError`	If the selection is unknown or non-contiguous, or if a step before it has not run yet.

Examples:

from llm_annotator import load_pipeline_config, run_pipeline
config = load_pipeline_config("config.yaml")
dataset = run_pipeline(config)
only = run_pipeline(config, selected=["judge"])

View source on GitHub: src/llm_annotator/pipeline.py lines 542–813

main ¶

main(args: list[str] | None = None) -> None

Run an annotation pipeline described by a JSON or YAML config file.

A config that does not load is reported as one error: line per problem, and the process exits with status 2. --debug keeps the traceback instead. An error raised while the pipeline runs keeps its traceback either way.

Parameters:

Name	Type	Description	Default
`args`	`list[str] \| None`	Optional argument list; defaults to `sys.argv`.	`None`

View source on GitHub: src/llm_annotator/pipeline.py lines 1362–1373