BPE

trimbed.backends.bpe

Selection constraints for byte-pair-encoding tokenizers (GPT-2, Qwen, Llama, ...).

BpeBackend

Bases: VocabBackend

Keeps a BPE vocabulary encodable and its merge chains intact.

structural_tokens

structural_tokens(spec: TokenizerSpec) -> set[str]

Return the unk token plus, for byte-level BPE, the whole byte alphabet.

Parameters:

Name	Type	Description	Default
spec	TokenizerSpec	The tokenizer being trimmed.	required

Returns:

Type	Description
set[str]	Token strings that may never be dropped. For codefuse-ai/F2LLM-v2-160M that is the 256 alphabet characters ("!", "Ġ" for a space, "Ċ" for a newline, ...) and nothing else, since it has no unknown token.

View source on GitHub: src/trimbed/backends/bpe.py lines 22–37

dependencies

dependencies(
    spec: TokenizerSpec,
) -> dict[int, tuple[int, ...]]

Map each merged token to the pair it is assembled from.

codefuse-ai/F2LLM-v2-160M builds "Ġthe" by merging "Ġth" with "e", so dropping "Ġth" leaves "Ġthe" in the vocabulary but unreachable and the text quietly tokenizes into characters instead. The selector uses this mapping to pull in whatever a kept token is built from.

Parameters:

Name	Type	Description	Default
spec	TokenizerSpec	The tokenizer being trimmed.	required

Returns:

Type	Description
dict[int, tuple[int, ...]]	A mapping of merged token id to its two parent ids, e.g. Ġthe -> (Ġth, e) and Ġworld -> (Ġw, orld). That is 151,387 entries for codefuse-ai/F2LLM-v2-160M, one per merge rule whose three tokens all survive.

View source on GitHub: src/trimbed/backends/bpe.py lines 39–63