Presets

trimbed.presets

Named sets of tokens a user can insist on keeping.

Some of these presets are functional and can take parameters so users can for instance write script:Latin to keep all tokens whose letters are Latin script.

Note that tokens are never "added", only the intersection of the preset and what is already in the vocabulary would be kept.

PresetInfo dataclass

PresetInfo(name: str, always_kept: bool, summary: str)

What trimbed presets prints about one preset.

register_preset

register_preset(
    name: str, always_kept: bool = False
) -> Callable[[PresetFn], PresetFn]

Register a preset and specify whether the trim always includes the preset's tokens.

Parameters:

Name	Type	Description	Default
name	str	The name users will write in selection.keep_presets, e.g. "digits".	required
always_kept	bool	Whether the preset resolves to tokens that TokenizerSpec.structural_ids already protects, so the trim keeps them whether or not the preset is named. True for "special_tokens" and the like.	False

Returns:

Type	Description
Callable[[PresetFn], PresetFn]	A decorator that registers the function and returns it unchanged.

Raises:

Type	Description
ValueError	If the name is already registered or contains the separator.

View source on GitHub: src/trimbed/presets.py lines 47–74

register_parametrised_preset

register_parametrised_preset(
    prefix: str,
) -> Callable[[ParametrisedPresetFn], ParametrisedPresetFn]

Register a preset family addressed as prefix:argument, such as script:Latin.

Parameters:

Name	Type	Description	Default
prefix	str	The part before the colon, e.g. "script", which users then write as script:Latin or script:Cyrillic.	required

Returns:

Type	Description
Callable[[ParametrisedPresetFn], ParametrisedPresetFn]	A decorator that registers the function and returns it unchanged.

Raises:

Type	Description
ValueError	If the prefix is already registered.

View source on GitHub: src/trimbed/presets.py lines 77–97

describe_presets

describe_presets() -> tuple[PresetInfo, ...]

Describe every registered preset, plain ones first and parametrised families last.

Returns:

Type	Description
tuple[PresetInfo, ...]	One entry per preset, carrying the name, whether the trim keeps those tokens anyway, and the first line of the preset function's docstring.

View source on GitHub: src/trimbed/presets.py lines 100–114

render_presets

render_presets() -> str

Render every registered preset as an aligned table, the structural ones first.

Returns:

Type	Description
str	The table that trimbed presets prints.

View source on GitHub: src/trimbed/presets.py lines 117–136

available_presets

available_presets() -> tuple[str, ...]

Return every registered preset name, with prefix:... for the parametrised ones.

E.g. ("added_tokens", "alphanumeric", "ascii_letters", ..., "unk", "whitespace", "script:...").

View source on GitHub: src/trimbed/presets.py lines 152–158

resolve_preset

resolve_preset(name: str, spec: TokenizerSpec) -> set[str]

Resolve one preset name against a tokenizer.

Parameters:

Name	Type	Description	Default
name	str	A registered preset name, optionally prefix:argument, e.g. "digits" or "script:Latin".	required
spec	TokenizerSpec	The tokenizer being trimmed.	required

Returns:

Type	Description
set[str]	The tokens the preset selects, all guaranteed to exist in spec. Sizes vary enormously with the preset: against codefuse-ai/F2LLM-v2-160M, digits selects 10 tokens, byte_alphabet 256, single_characters 18,747 and ascii_letters 70,096 of the 151,669.

Raises:

Type	Description
KeyError	If no preset matches the name.

View source on GitHub: src/trimbed/presets.py lines 161–183

resolve_presets

resolve_presets(
    names: Iterable[str], spec: TokenizerSpec
) -> dict[str, set[str]]

Resolve several presets, keeping the result attributable per preset.

Parameters:

Name	Type	Description	Default
names	Iterable[str]	Preset names to resolve, e.g. ["special_tokens", "digits"].	required
spec	TokenizerSpec	The tokenizer being trimmed.	required

Returns:

Type	Description
dict[str, set[str]]	A mapping of preset name to the tokens it selected. Keeping it attributable per preset is what lets the report say preset:digits=10 rather than one undifferentiated total.

View source on GitHub: src/trimbed/presets.py lines 186–198