trimbed compare¶

trimbed.cli.compare_tokenizers ¶

Diff two tokenizers and print what the difference between them amounts to.

Point it at a checkpoint and a trimmed version of it to find out what the trim actually did: whether the smaller vocabulary is a subset of the larger one and still in the original order, whether the added, special and post-processor tokens survived, which presets and Unicode scripts were kept or gutted, and how much longer the same text encodes. Neither model's weights are read and nothing is written unless you ask for JSON.

trimbed compare clips/e5-small-trm-nl clips/e5-small-trm
trimbed compare bert-base-multilingual-cased trimmed/bert --text-file dutch.txt
trimbed compare base/ trimmed/ --preset script:Latin --preset script:Cyrillic -o diff.json

The comparison is directional: everything is counted against the first tokenizer, so LATIN 45,102/60,003 means the base had 60,003 Latin tokens and 45,102 of them survive.

run ¶

run(
    base: str,
    other: str,
    presets: list[str] | None = None,
    texts: list[str] | None = None,
    text_file: str | None = None,
    examples: int = 10,
    output: str | None = None,
    trust_remote_code: bool = False,
    verbose: bool = False,
    quiet: bool = False,
) -> None

Compare two tokenizers and print the diff.

Parameters:

Name	Type	Description	Default
`base`	`str`	Hub model id or local path of the original tokenizer.	required
`other`	`str`	Hub model id or local path of the tokenizer to compare against it.	required
`presets`	`list[str] \| None`	Extra preset names to resolve, which is how the parametrised ones are reached, e.g. `["script:Latin"]`.	`None`
`texts`	`list[str] \| None`	Sample texts to encode with both tokenizers.	`None`
`text_file`	`str \| None`	Text file holding one further sample text per line.	`None`
`examples`	`int`	How many removed and introduced tokens to quote.	`10`
`output`	`str \| None`	JSON file to write the full report to.	`None`
`trust_remote_code`	`bool`	Allow tokenizer code shipped with either checkpoint.	`False`
`verbose`	`bool`	Emit debug logging.	`False`
`quiet`	`bool`	Only emit warnings and errors.	`False`

View source on GitHub: src/trimbed/cli/compare_tokenizers.py lines 28–68

add_arguments ¶

add_arguments(parser: ArgumentParser) -> None

Add the comparison arguments to parser.

Parameters:

Name	Type	Description	Default
`parser`	`ArgumentParser`	The `trimbed compare` subparser to populate.	required

View source on GitHub: src/trimbed/cli/compare_tokenizers.py lines 78–109