Hub restore¶

llm_annotator.hub ¶

Restore the Hugging Face Hub backups that an annotation run pushes.

restore_progress_from_hub ¶

restore_progress_from_hub(
    *,
    hub_id: str,
    output_dir: str | Path,
    task_prefix: str = "",
    idx_column: str = "idx",
    force: bool = False,
) -> Path

Download a run's Hub progress backup into its output directory.

The branch <task_prefix>progress_backup of the dataset repository holds the JSONL progress files and the selection record of the run. The progress files are written to <output_dir>/<task_prefix>progress_backup/ and the record next to that directory, which is where prepare_data and run_annotation read them. Run this before the annotation on a machine that has no local output directory, so the run resumes instead of annotating every row again.

A progress directory that already holds .jsonl files is refused unless force is set. With force the rows are merged per sample id and a local row wins, so no finished row is lost. Merging on the id rather than on the file length is what makes this safe for a run whose files hold different rows under the same name: a local progress_backup_0.jsonl with the ids 50-99 (a server pool finishes batches out of order) and a Hub file of the same name with the ids 0-49 give one file with the ids 0-99, where the longer of the two files would drop 50 finished rows. A row that a run deleted on purpose (retry_errors) comes back when the backup still holds it, so apply retry_errors after the restore.

Parameters:

Name	Type	Description	Default
`hub_id`	`str`	The dataset repository that the run backs up to.	required
`output_dir`	`str \| Path`	The annotator's output directory. For a pipeline step this is `<output_dir>/<NN>-<name>/annotate/`.	required
`task_prefix`	`str`	The task prefix of the run, which names both the branch and the local directory. For a pipeline step this is `<name>_`.	`''`
`idx_column`	`str`	Column that holds the sample id. Only read when rows are merged into existing progress files.	`'idx'`
`force`	`bool`	Whether to merge into a progress directory that already holds files.	`False`

Returns:

Type	Description
`Path`	The local progress directory that the rows were written to.

Raises:

Type	Description
`FileExistsError`	If the progress directory already holds `.jsonl` files and `force` is not set.
`ValueError`	If the repository has no backup branch for this task.

Examples:

restore_progress_from_hub(
    hub_id="me/my-dataset",
    output_dir="outputs/run",
    task_prefix="sentiment_",
)

View source on GitHub: src/llm_annotator/hub.py lines 123–244

main ¶

main(args: list[str] | None = None) -> None

Restore a run's Hub progress backup into its local output directory.

Parameters:

Name	Type	Description	Default
`args`	`list[str] \| None`	Optional argument list; defaults to `sys.argv`.	`None`

View source on GitHub: src/llm_annotator/hub.py lines 303–310