trimbed count¶

trimbed.cli.count_tokens ¶

Count corpus token frequencies once and cache them to JSON.

Counting is the expensive part of a trimming run and it does not depend on the selection policy, so do it once here and point corpus.counts_cache at the output to reuse it across selection experiments.

trimbed count --config my_config.yaml -o counts.json
trimbed count --config my_config.yaml -o counts.json         corpus.batch_size=4000 corpus.num_proc=8

Only the corpus half of the config matters here. The selection and embedding sections are still validated, but they are not used.

run ¶

run(
    output: str,
    config: str | None = None,
    model: str | None = None,
    overrides: list[str] | None = None,
    verbose: bool = False,
    quiet: bool = False,
) -> None

Count the configured corpus and write the frequencies to a cache file.

Parameters:

Name	Type	Description	Default
`output`	`str`	JSON file to write the counts to.	required
`config`	`str \| None`	Path to a YAML configuration file.	`None`
`model`	`str \| None`	Hub model id or local path, overriding the config.	`None`
`overrides`	`list[str] \| None`	`key=value` strings applied on top of the config and flags.	`None`
`verbose`	`bool`	Emit debug logging.	`False`
`quiet`	`bool`	Only emit warnings and errors.	`False`

Raises:

Type	Description
`ValueError`	If the resolved config names no datasets to count over.

View source on GitHub: src/trimbed/cli/count_tokens.py lines 25–61

add_arguments ¶

add_arguments(parser: ArgumentParser) -> None

Add the counting arguments to parser.

Parameters:

Name	Type	Description	Default
`parser`	`ArgumentParser`	The `trimbed count` subparser to populate.	required

View source on GitHub: src/trimbed/cli/count_tokens.py lines 70–85