Tokenizer trimming¶

trimbed.tokenizer_trim ¶

Applying a trim to a tokenizer, via skeletoken's typed data model.

LEGACY_VOCAB_FILES `module-attribute` ¶

LEGACY_VOCAB_FILES = (
    "vocab.json",
    "vocab.txt",
    "merges.txt",
    "added_tokens.json",
    "spiece.model",
    "sentencepiece.bpe.model",
    "source.spm",
    "target.spm",
)

Vocabulary files a fast tokenizer may write alongside tokenizer.json.

They are regenerated from the backend on save, so a stale copy left in the output directory would describe the untrimmed vocabulary.

TrimmedTokenizer `dataclass` ¶

TrimmedTokenizer(
    tokenizer: PreTrainedTokenizerFast,
    spec: TokenizerSpec,
    remap: IdRemap,
)

A trimmed tokenizer and the mapping that produced it.

Attributes:

Name	Type	Description
`tokenizer`	`PreTrainedTokenizerFast`	The reloaded `transformers` fast tokenizer, read back from disk rather than constructed in memory, which is what proves it loads.
`spec`	`TokenizerSpec`	A spec over the trimmed tokenizer, whose `source` is the original's with `" (trimmed)"` added to disambiguate.
`remap`	`IdRemap`	The old-id to new-id mapping that was applied.

build_trimmed_model ¶

build_trimmed_model(
    spec: TokenizerSpec, kept_ids: Iterable[int]
) -> TokenizerModel

Remove everything outside kept_ids from the tokenizer.

skeletoken performs the surgery and validates the result: it compacts the ids, filters the merge table, prunes the added tokens and renumbers every id another component hard-codes (the post-processor's special_tokens, padding.pad_id, Unigram's unk_id, maybe others...)

Parameters:

Name	Type	Description	Default
`spec`	`TokenizerSpec`	The tokenizer being trimmed.	required
`kept_ids`	`Iterable[int]`	Ids that survive, e.g. the 461 that a preset-and-chat-template selection leaves of codefuse-ai/F2LLM-v2-160M's 151,669.	required

Returns:

Type	Description
`TokenizerModel`	The trimmed skeletoken model, renumbered from zero.

View source on GitHub: src/trimbed/tokenizer_trim.py lines 58–77

trim_tokenizer ¶

trim_tokenizer(
    tokenizer: PreTrainedTokenizerFast,
    spec: TokenizerSpec,
    kept_ids: Iterable[int],
) -> TrimmedTokenizer

Apply a trim to a fast tokenizer and return the trimmed result.

The trimmed tokenizer config is round-tripped through save_pretrained and AutoTokenizer.from_pretrained rather than through skeletoken's to_transformers, because the latter constructs a fresh tokenizer object and so loses what lives outside tokenizer.json: the chat template and the rest of tokenizer_config.json. Reloading also proves the result actually loads.

Parameters:

Name	Type	Description	Default
`tokenizer`	`PreTrainedTokenizerFast`	The original fast tokenizer.	required
`spec`	`TokenizerSpec`	A spec over that tokenizer.	required
`kept_ids`	`Iterable[int]`	Ids that survive the trim.	required

Returns:

Type	Description
`TrimmedTokenizer`	The trimmed tokenizer, its spec, and the remap that was applied.

Raises:

Type	Description
`RuntimeError`	If the reloaded tokenizer does not match the trimmed model.

View source on GitHub: src/trimbed/tokenizer_trim.py lines 106–150