trimbed trim¶

trimbed.cli.trim_vocab ¶

Trim a tokenizer's vocabulary down to a useful subset, and optionally its model with it.

The run is described by a YAML config, documented field by field in the configuration guide. Three layers stack, later winning over earlier: the YAML file, the typed flags below, and the trailing key=value positionals. The typed flags cover the knobs worth tuning between runs, and the positionals reach every remaining field without needing a flag for each one.

trimbed trim --config my_config.yaml
trimbed trim --config my_config.yaml --dry-run         selection.top_k=30000 corpus.batch_size=2000

A config is not required: --model plus must-keep rules is enough to trim without a corpus, which is the fastest way to see what the machinery does.

trimbed trim --model google-bert/bert-base-multilingual-cased         --keep-preset alphanumeric --output-dir trimmed/bert --no-trim-model

run ¶

run(
    config: str | None = None,
    model: str | None = None,
    output_dir: str | None = None,
    coverage: float | None = None,
    top_k: int | None = None,
    min_count: int | None = None,
    max_vocab_size: int | None = None,
    keep_presets: list[str] | None = None,
    keep_tokens: list[str] | None = None,
    keep_texts: list[str] | None = None,
    no_keep_chat_template: bool = False,
    no_trim_model: bool = False,
    no_verify: bool = False,
    no_verify_model: bool = False,
    trust_remote_code: bool = False,
    overwrite: bool = False,
    dry_run: bool = False,
    overrides: list[str] | None = None,
    verbose: bool = False,
    quiet: bool = False,
) -> None

Resolve the configuration, run the trimming pipeline and print its report.

TrimConfig.with_overrides drops None values, so an argument that was not supplied leaves the config alone. The boolean flags therefore map to their meaningful value or to None, never to the config default. That is why a flag turning something off is spelled --no-... and one turning something on is not.

Parameters:

Name	Type	Description	Default
`config`	`str \| None`	Path to a YAML configuration file.	`None`
`model`	`str \| None`	Hub model id or local path, overriding the config.	`None`
`output_dir`	`str \| None`	Directory to write the trimmed artefacts to.	`None`
`coverage`	`float \| None`	Keep tokens covering this fraction of corpus occurrences.	`None`
`top_k`	`int \| None`	Keep at most this many corpus-derived tokens.	`None`
`min_count`	`int \| None`	Keep tokens seen at least this many times.	`None`
`max_vocab_size`	`int \| None`	Hard cap on the final vocabulary size.	`None`
`keep_presets`	`list[str] \| None`	Must-keep preset names.	`None`
`keep_tokens`	`list[str] \| None`	Literal tokens to keep.	`None`
`keep_texts`	`list[str] \| None`	Texts that must keep encoding the way they do now.	`None`
`no_keep_chat_template`	`bool`	Do not keep the tokens the chat template's own words need.	`False`
`no_trim_model`	`bool`	Trim only the tokenizer, not the model.	`False`
`no_verify`	`bool`	Skip the round-trip verification pass.	`False`
`no_verify_model`	`bool`	Skip running both models and comparing their outputs.	`False`
`trust_remote_code`	`bool`	Allow custom code shipped with the checkpoint.	`False`
`overwrite`	`bool`	Allow writing into a non-empty output directory.	`False`
`dry_run`	`bool`	Select and report without writing anything.	`False`
`overrides`	`list[str] \| None`	`key=value` strings applied on top of the config and flags.	`None`
`verbose`	`bool`	Emit debug logging.	`False`
`quiet`	`bool`	Only emit warnings and errors.	`False`

View source on GitHub: src/trimbed/cli/trim_vocab.py lines 29–102

add_arguments ¶

add_arguments(parser: ArgumentParser) -> None

Add the trim arguments to parser.

Parameters:

Name	Type	Description	Default
`parser`	`ArgumentParser`	The `trimbed trim` subparser to populate.	required

View source on GitHub: src/trimbed/cli/trim_vocab.py lines 112–167