trimbed inspect¶

trimbed.cli.inspect_tokenizer ¶

Print a tokenizer's shape and whether trimbed can trim it.

Run this before a trimming job: it reports the backend family (BPE, WordPiece, Unigram, WordLevel), the vocabulary size, how many tokens are added or special (those are structural and never removed), whether it ships a chat template, and whether a backend is registered for the family at all.

trimbed inspect --model codefuse-ai/F2LLM-v2-160M
trimbed inspect --config my_config.yaml

Nothing is loaded beyond the tokenizer and nothing is written. The model weights are never touched.

run ¶

run(
    config: str | None = None,
    model: str | None = None,
    verbose: bool = False,
    quiet: bool = False,
) -> None

Describe the tokenizer as JSON on stdout.

Parameters:

Name	Type	Description	Default
`config`	`str \| None`	Path to a YAML configuration file.	`None`
`model`	`str \| None`	Hub model id or local path, overriding the config.	`None`
`verbose`	`bool`	Emit debug logging.	`False`
`quiet`	`bool`	Only emit warnings and errors.	`False`

View source on GitHub: src/trimbed/cli/inspect_tokenizer.py lines 27–43

add_arguments ¶

add_arguments(parser: ArgumentParser) -> None

Add the inspection arguments to parser.

Parameters:

Name	Type	Description	Default
`parser`	`ArgumentParser`	The `trimbed inspect` subparser to populate.	required

View source on GitHub: src/trimbed/cli/inspect_tokenizer.py lines 52–65