LLM Annotator¶

LLM Annotator runs a large language model over a dataset and writes the answers back as columns. A run is resumable: every finished sample is appended to a JSONL progress file, so a job that crashes, times out or is preempted continues where it stopped instead of starting over.

Four providers share one interface:

VLLMOfflineClient for in-process vLLM (vllm_offline).
VLLMOnlineClient for a vLLM server, one or many (vllm_online).
OpenAIClient for OpenAI-compatible APIs (openai).
ClaudeClient for the Anthropic API (claude).

Choosing a provider says which one fits the hardware you have. Provider setup lists the extras, the authentication variables and the vLLM tuning knobs.

Install¶

uv add llm-annotator

or

pip install llm-annotator

A provider needs its extra:

uv add "llm-annotator[vllm]"       # vllm_offline
uv add "llm-annotator[openai]"     # openai and vllm_online
uv add "llm-annotator[anthropic]"  # claude

The online vLLM client speaks the OpenAI protocol, so it takes the openai extra rather than the much heavier vllm one, which is only needed where the model weights are loaded. See Provider setup.

Quickstart¶

Describe the run in one YAML (or JSON) file and start it. No Python needed.

my-pipeline.yaml

output_dir: outputs/imdb-sentiment

dataset:
  name: stanfordnlp/imdb
  split: test
  max_num_samples: 20

client:
  provider: vllm_offline
  model: HuggingFaceTB/SmolLM2-135M-Instruct

steps:
  - name: sentiment
    prompt: "Classify the sentiment: {text}"

llm-annotate my-pipeline.yaml

The result is a dataset with a sentiment_response column next to the original text, written to outputs/imdb-sentiment/final/.

A config can hold several steps that run in order, each annotating what the previous one produced. Annotating from a config file is the key reference, and examples/pipeline-qa/ in the repository is a complete two-step example.

The same run from Python:

from llm_annotator import Annotator, VLLMOfflineClient

client = VLLMOfflineClient(model="HuggingFaceTB/SmolLM2-135M-Instruct")

with Annotator(client=client, verbose=True) as anno:
    ds = anno.annotate_dataset(
        output_dir="outputs/imdb-sentiment",
        prompt_template="Classify the sentiment: {text}",
        dataset_name="stanfordnlp/imdb",
        dataset_split="test",
        max_num_samples=20,
    )

Or build a dataset from scratch instead of annotating one:

from llm_annotator import Annotator, OpenAIClient

client = OpenAIClient(model="gpt-4o-mini")

with Annotator(client=client) as anno:
    ds = anno.generate_dataset(
        output_dir="outputs/generated-qa",
        prompts="Write a short geography quiz question with answer.",
        max_num_samples=200,
    )

Where to go next¶

Choosing a provider: one GPU, many GPUs or none.
Annotating from a config file: every config key, multi-step pipelines, the llm-annotate command line.
Python API guide: the staged prepare_data plus run_annotation workflow, several tasks in one output directory, errors and retries, a pool of vLLM servers.
Growing a run: resume a run, raise the sample cap, edit a prompt mid-run.
SLURM: submit a config as one job chain per step.
Troubleshooting: what an error message means and what to do about it.
Migrating from 0.16: everything that changed since the last release.

The "User API" section of the navigation is generated from the docstrings of what the guides above use; "Internals" covers the modules those build on.