Pipeline runner¶

llm_annotator.pipeline ¶

Sequential executor and CLI for config-driven annotation pipelines.

run_pipeline walks the steps of a PipelineConfig in order, handing the dataset each step produces to the next one. Nothing here re-implements annotation: a step is one ordinary annotate_dataset call, so prompt templating, JSONL progress checkpoints, resumption, retries and Hub backups behave exactly as they do when the library is used directly.

Two things make a pipeline more than a loop:

Every step owns a subdirectory of output_dir and a task_prefix, so its internal columns and artifacts cannot collide with another step's.
A finished step writes its result to <step-dir>/output. On a re-run that snapshot is loaded and the step is skipped, so a pipeline that dies in step three does not repeat steps one and two.

main is the llm-annotate console entry point.

STEP_OUTPUT_SUBDIR `module-attribute` ¶

STEP_OUTPUT_SUBDIR = 'output'

Name of the subdirectory holding a finished step's dataset.

Its presence is what marks a step as done.

STEP_ANNOTATE_SUBDIR `module-attribute` ¶

STEP_ANNOTATE_SUBDIR = 'annotate'

Name of the subdirectory handed to the annotator as its output_dir.

It is one level below the step directory because _post_annotate writes a dataset into the root of whatever output_dir it is given.

run_pipeline ¶

run_pipeline(
    config: PipelineConfig,
    selected: Sequence[str] | None = None,
) -> Dataset

Run a pipeline, or part of one, and return the resulting dataset.

Steps share one live client whenever their provider, model and constructor settings match, so a pipeline that uses the same local model twice loads it only once. The client is always released before returning, including on failure.

Passing selected runs only those steps. Earlier steps must already have finished: their saved output is loaded as the input, which is what lets a scheduler run one step per job while keeping a single config file as the source of truth.

Parameters:

Name	Type	Description	Default
`config`	`PipelineConfig`	The validated pipeline configuration.	required
`selected`	`Sequence[str] \| None`	Names of the steps to run. `None` runs all of them. The names must form a contiguous run of the pipeline.	`None`

Returns:

Type	Description
`Dataset`	The dataset produced by the last step that ran.

Raises:

Type	Description
`ValueError`	If the selection is unknown or non-contiguous, or if a step before it has not run yet.

Examples:

from llm_annotator import load_pipeline_config, run_pipeline
config = load_pipeline_config("config.yaml")
dataset = run_pipeline(config)
only = run_pipeline(config, selected=["judge"])

View source on GitHub: src/llm_annotator/pipeline.py lines 339–489

main ¶

main(args: list[str] | None = None) -> None

Run an annotation pipeline described by a JSON or YAML config file.

Parameters:

Name	Type	Description	Default
`args`	`list[str] \| None`	Optional argument list; defaults to `sys.argv`.	`None`

View source on GitHub: src/llm_annotator/pipeline.py lines 609–723