Selection

trimbed.selection

Deciding which token ids survive the trim and recording the reason.

STRUCTURAL module-attribute

STRUCTURAL = 'structural'

Provenance label for a token that was never eligible for removal.

CORPUS module-attribute

CORPUS = 'corpus'

Provenance label for a token the corpus frequency ranking kept.

DEPENDENCY module-attribute

DEPENDENCY = 'dependency'

Provenance label for a token another kept token is built out of.

Selection dataclass

Selection(
    kept_ids: set[int] = set(),
    structural_ids: set[int] = set(),
    provenance: dict[int, set[str]] = (
        lambda: defaultdict(set)
    )(),
    dropped_requested: dict[int, set[str]] = dict(),
    unknown_tokens: list[str] = list(),
)

The outcome of applying a SelectionConfig.

Attributes:

Name	Type	Description
kept_ids	set[int]	Token ids that survive, as a set.
structural_ids	set[int]	Ids that were never eligible for removal.
provenance	dict[int, set[str]]	For each kept id, the reasons it was kept, e.g. {151645: {"structural", "chat_template"}, 9707: {"corpus"}}. The labels are "structural", "corpus", "dependency", "text", "chat_template", "explicit_token", "explicit_id", or a prefixed "preset:digits", "pattern:^Ġ", "file:keep.txt". Tracking this is what lets the report say which rule saved each token, rather than only how many survived in total.
dropped_requested	dict[int, set[str]]	Ids that were requested but that the size cap had to remove anyway, mapped to the reasons they had been requested.
unknown_tokens	list[str]	Requested token strings that this tokenizer does not contain, e.g. ["<|custom|>"] when a keep-list was written for another checkpoint.

__len__

__len__() -> int

Return the size of the trimmed vocabulary.

View source on GitHub: src/trimbed/selection.py lines 59–61

counts_by_reason

counts_by_reason() -> dict[str, int]

Return how many kept ids carry each provenance label.

An id kept for several reasons is counted under each of them, so the values sum to more than len(self).

A preset-and-chat-template trim of codefuse-ai/F2LLM-v2-160M reports {"chat_template": 88, "dependency": 101, "preset:byte_alphabet": 256, "preset:digits": 10, "preset:special_tokens": 14, "structural": 282, "text": 8} for 461 kept tokens.

View source on GitHub: src/trimbed/selection.py lines 63–77

select_tokens

select_tokens(
    spec: TokenizerSpec,
    counts: CorpusCounts | None,
    config: SelectionConfig,
) -> Selection

Decide which token ids survive the trim.

The rule is the union of structural, requested and corpus, closed over the backend's token dependencies, followed by the max_vocab_size cap if one was asked for. Presets and explicit lists therefore act as a floor on the vocabulary rather than as a filter on it.

Parameters:

Name	Type	Description	Default
spec	TokenizerSpec	The tokenizer being trimmed.	required
counts	CorpusCounts | None	Corpus statistics, or None when selecting from explicit sources only.	required
config	SelectionConfig	The selection settings.	required

Returns:

Type	Description
Selection	The kept ids together with the provenance of each. Keeping the special tokens, the byte alphabet, the digits, the chat template's words and one Dutch sentence leaves 461 of codefuse-ai/F2LLM-v2-160M's 151,669 tokens, 282 of them structural.

View source on GitHub: src/trimbed/selection.py lines 343–394