Unigram¶

trimbed.backends.unigram ¶

Selection constraints for Unigram tokenizers (XLM-R, mT5, ALBERT, ...).

Keeps a Unigram vocabulary usable.

Unigram scores whole candidate pieces independently rather than composing them from merges.

structural_tokens(spec: TokenizerSpec) -> set[str]

Return the unk token plus any byte-fallback tokens the model relies on.

Parameters:

Name	Type	Description	Default
`spec`	`TokenizerSpec`	The tokenizer being trimmed.	required

Returns:

Type	Description
`set[str]`	Token strings that may never be dropped, e.g. `{"<unk>"}` for google-t5/t5-small and FacebookAI/xlm-roberta-base.