Backend base¶

trimbed.backends.base ¶

The base interface for different tokenizer families.

We luckily can rely on skeletoken to do the vocabulary surgery for every tokenizer family, so these adapters only carry what a trimmer has to know and a serialiser does not: which tokens can never be dropped, and which tokens depend on which.

VocabBackend ¶

Bases: ABC

Describes the structural constraints of one tokenizers model type.

Subclasses answer which tokens can never be removed without breaking encoding outright and which tokens can only be produced if some other token survives.

model_type `class-attribute` ¶

model_type: str

Value of model.type in tokenizer.json that this adapter handles.

E.g. "BPE" for codefuse-ai/F2LLM-v2-160M, "WordPiece" for google-bert/bert-base-cased and "Unigram" for google-t5/t5-small.

structural_tokens ¶

structural_tokens(spec: TokenizerSpec) -> set[str]

Return tokens that must survive or the tokenizer stops working.

The default covers the unknown token. Byte-level backends extend it with the byte alphabet.

Parameters:

Name	Type	Description	Default
`spec`	`TokenizerSpec`	The tokenizer being trimmed.	required

Returns:

Type	Description
`set[str]`	Token strings that may never be dropped. E.g. `{"[UNK]"}` for google-bert/bert-base-cased, and an empty set for codefuse-ai/F2LLM-v2-160M, which declares no unknown token at all.

View source on GitHub: src/trimbed/backends/base.py lines 32–47

dependencies ¶

dependencies(
    spec: TokenizerSpec,
) -> dict[int, tuple[int, ...]]

Return which other tokens each token needs in order to stay reachable.

Keeping a token while dropping something it is assembled from leaves it in the vocabulary but unreachable, so text silently fragments. Backends with no such structure return an empty mapping.

Parameters:

Name	Type	Description	Default
`spec`	`TokenizerSpec`	The tokenizer being trimmed.	required

Returns:

Type	Description
`dict[int, tuple[int, ...]]`	A mapping of token id to the ids it directly depends on. Empty here, since BPE is the only family that builds tokens out of other tokens. See `BpeBackend.dependencies` for how it fills this in.

View source on GitHub: src/trimbed/backends/base.py lines 49–65

repr ¶

__repr__() -> str

Return a short debug representation.

View source on GitHub: src/trimbed/backends/base.py lines 83–85