Utils

llm_annotator.utils

get_hash

get_hash(text: str) -> str

Compute a SHA256 hash for a given text string.

Parameters:

Name	Type	Description	Default
text	str	The input string to hash.	required

Returns:

Type	Description
str	A 64-character hexadecimal SHA256 digest.

Examples:

len(get_hash("hello"))
# 64
get_hash("hello") == get_hash("hello")
# True
get_hash("hello") == get_hash("world")
# False

View source on GitHub: src/llm_annotator/utils.py lines 19–36

dataset_signature

dataset_signature(
    dataset: Dataset, num_probe_rows: int = 64
) -> str

Compute a content signature of a dataset without reading all of it.

The signature is a SHA256 hash of the row count, the column names and up to num_probe_rows evenly spaced rows. It depends on the content only, so it is the same in every process and for every version of datasets. A change in the number of rows always changes it. An edit of a row that is not probed does not.

Parameters:

Name	Type	Description	Default
dataset	Dataset	The dataset to describe.	required
num_probe_rows	int	Maximum number of rows that are hashed.	64

Returns:

Type	Description
str	A 64-character hexadecimal SHA256 digest.

Examples:

from datasets import Dataset
first = Dataset.from_dict({"text": ["a", "b"]})
same = Dataset.from_dict({"text": ["a", "b"]})
longer = Dataset.from_dict({"text": ["a", "b", "c"]})
dataset_signature(first) == dataset_signature(same)
# True
dataset_signature(first) == dataset_signature(longer)
# False

View source on GitHub: src/llm_annotator/utils.py lines 39–82

remove_empty_jsonl_files

remove_empty_jsonl_files(pdout: Path) -> list[Path]

Remove any empty .jsonl files in the given directory.

Parameters:

Name	Type	Description	Default
pdout	Path	Output directory path to clean up.	required

Returns:

Type	Description
list[Path]	A list of removed files.

View source on GitHub: src/llm_annotator/utils.py lines 85–100

drop_jsonl_rows

drop_jsonl_rows(
    pdout: Path,
    should_drop: Callable[[dict[str, Any]], bool],
) -> list[dict[str, Any]]

Remove the rows that match a predicate from every .jsonl file in a directory.

A file that loses rows is written to a temporary file that then replaces it, so a crash leaves either the old or the new file. A line that is not valid JSON is kept as it is.

Parameters:

Name	Type	Description	Default
pdout	Path	Directory that holds the *.jsonl files.	required
should_drop	Callable[[dict[str, Any]], bool]	Called with each parsed row. True removes the row.	required

Returns:

Type	Description
list[dict[str, Any]]	The removed rows.

Examples:

import tempfile
pdout = Path(tempfile.mkdtemp())
_ = (pdout / "rows.jsonl").write_text('{"idx": 0}\n{"idx": 1}\n')
drop_jsonl_rows(pdout, lambda row: row["idx"] == 1)
# [{'idx': 1}]
(pdout / "rows.jsonl").read_text()
# '{"idx": 0}\n'

View source on GitHub: src/llm_annotator/utils.py lines 103–148

ensure_returns_bool

ensure_returns_bool(
    func: Callable[[dict[str, Any]], Any],
    sample: dict[str, Any],
) -> bool

Call a user validation hook and check that it returned a boolean.

Parameters:

Name	Type	Description	Default
func	Callable[[dict[str, Any]], Any]	The hook, called with one sample.	required
sample	dict[str, Any]	The sample to pass to func.	required

Returns:

Type	Description
bool	The boolean result returned by func.

Raises:

Type	Description
TypeError	If func does not return a boolean.

Examples:

ensure_returns_bool(lambda sample: bool(sample), {"a": 1})
# True

View source on GitHub: src/llm_annotator/utils.py lines 151–175

ensure_returns_dict

ensure_returns_dict(
    func: Callable[[dict[str, Any]], Any],
    sample: dict[str, Any],
) -> dict[str, Any]

Call a user postprocessing hook and check that it returned a dict.

Parameters:

Name	Type	Description	Default
func	Callable[[dict[str, Any]], Any]	The hook, called with one sample.	required
sample	dict[str, Any]	The sample to pass to func.	required

Returns:

Type	Description
dict[str, Any]	The dictionary result returned by func.

Raises:

Type	Description
TypeError	If func does not return a dictionary.

Examples:

ensure_returns_dict(lambda sample: {**sample, "b": 2}, {"a": 1})
# {'a': 1, 'b': 2}

View source on GitHub: src/llm_annotator/utils.py lines 178–202

get_lib_versions

get_lib_versions() -> dict[str, str]

Get the versions of key dependencies.

View source on GitHub: src/llm_annotator/utils.py lines 205–229

get_hf_username

get_hf_username() -> str | None

Get the Hugging Face username of the current user, if logged in. Otherwise, return None.

Returns:

Type	Description
str | None	The Hugging Face username, or None if not logged in.

View source on GitHub: src/llm_annotator/utils.py lines 232–245

extract_prompt_prefix

extract_prompt_prefix(prompt: str) -> str

Extract the prefix of a prompt up to the first {placeholder}, or the entire prompt if none exists.

Can return an empty string when the prompt starts with a {placeholder}. This is expected when using generate_dataset with fully variable prompts.

Parameters:

Name	Type	Description	Default
prompt	str	The full prompt string, optionally containing {field} placeholders.	required

Returns:

Type	Description
str	The substring before the first {placeholder}, or the entire prompt when
str	no placeholder is present.

Examples:

extract_prompt_prefix("Classify: {text}")
# 'Classify: '
extract_prompt_prefix("{text} is the input")
# ''
extract_prompt_prefix("No placeholders here")
# 'No placeholders here'

View source on GitHub: src/llm_annotator/utils.py lines 251–272

add_schema_additional_properties_false

add_schema_additional_properties_false(schema: Any) -> Any

Recursively set additionalProperties: false on all object schemas.

Claude requires this on every object type in the schema; without it the API returns a 400 error.

Parameters:

Name	Type	Description	Default
schema	Any	A JSON-schema dict (or any nested value).	required

Returns:

Type	Description
Any	A new schema dict with additionalProperties set to False on
Any	every sub-schema whose type is "object".

View source on GitHub: src/llm_annotator/utils.py lines 275–295