Tokenizer spec¶

trimbed.spec ¶

An inspectable view of a tokenizer, backed by skeletoken's typed data model.

Every Hugging Face (fast) tokenizer serialises to one JSON file (tokenizer.json), regardless of whether it is BPE, WordPiece, Unigram or WordLevel. skeletoken supplies a typed, validated model of that document, so here we only add what is needed for trimming: decoded surface forms for preset matching, and a lookup of the backend adapter that knows which tokens must never be dropped.

JINJA_CONSTRUCT `module-attribute` ¶

JINJA_CONSTRUCT = re.compile(
    "\\{[{%#].*?[}%#]\\}", re.DOTALL
)

{{ expression }}, {% statement %} and {# comment #}.

Everything a Jinja template evaluates rather than outputs literally. re.DOTALL makes . match newlines.

QUOTED_STRING `module-attribute` ¶

QUOTED_STRING = re.compile('\'([^\']*)\'|\\"([^\\"]*)\\"')

A quoted string inside a Jinja construct.

Role names reach the output through a comparison (message['role'] == 'user') rather than as literal text, so they only show up here.

CHAT_ROLES `module-attribute` ¶

CHAT_ROLES = (
    "system",
    "user",
    "assistant",
    "tool",
    "function",
    "developer",
)

Roles a ChatML-style template substitutes from the message without ever naming.

Such a template writes {{ message['role'] }} without being explicit about which roles are expected, so as a precaution the standard ones are kept even when the template does not mention them anywhere.

TokenizerSpec `dataclass` ¶

TokenizerSpec(
    tokenizer_model: TokenizerModel,
    source: str | None = None,
    chat_template: str | None = None,
)

A tokenizer, ready to be inspected and trimmed.

Attributes:

Name	Type	Description
`tokenizer_model`	`TokenizerModel`	The skeletoken model of the tokenizer.json document.
`source`	`str \| None`	Where the tokenizer came from (model id or path), for error messages. E.g. `"codefuse-ai/F2LLM-v2-160M"` or `"trimmed/f2llm-nl/tokenizer.json"`.
`chat_template`	`str \| None`	The Jinja chat template, when the tokenizer ships one. It lives in tokenizer_config.json rather than in tokenizer.json, so a spec built from a bare document does not have it.

model `property` ¶

model: Model

Return the typed model sub-object (BPE, WordPiece, Unigram or WordLevel).

model_type `property` ¶

model_type: str

Return the backend model type.

E.g. "BPE" for codefuse-ai/F2LLM-v2-160M, "WordPiece" for google-bert/bert-base-cased, "Unigram" for google-t5/t5-small.

backend `cached` `property` ¶

backend: VocabBackend

Return the registered adapter for this tokenizer's model type.

vocabulary `cached` `property` ¶

vocabulary: dict[str, int]

Return the token -> id map, added tokens included, since skeletoken folds those in.

E.g. for codefuse-ai/F2LLM-v2-160M this has 151,669 entries, holding ordinary tokens like "Ġthe" alongside added ones like "<|im_end|>" (id 151,645) with nothing to tell them apart here.

id_to_token `cached` `property` ¶

id_to_token: dict[int, str]

Return the id -> token map, e.g. {9707: "Hello", 1879: "Ġworld", ...}.

added_tokens `property` ¶

added_tokens: list[AddedToken]

Return the typed added_tokens entries.

E.g. 26 of them for codefuse-ai/F2LLM-v2-160M, starting at <|endoftext|> (id 151643), <|im_start|> (151644) and <|im_end|> (151645).

added_token_ids `cached` `property` ¶

added_token_ids: frozenset[int]

Return the ids of all added tokens, e.g. 26 ids for codefuse-ai/F2LLM-v2-160M.

special_token_ids `cached` `property` ¶

special_token_ids: frozenset[int]

Return the ids of added tokens flagged special.

Flagging happens in the added_tokens section of tokenizer.json (mirrored by added_tokens_decoder in tokenizer_config.json), where each added token carries a handful of flags, one of them "special": true. Qwen 3 marks "<|endoftext|>" and "<|im_end|>" as special but leaves "<tool_call>" and "<think>" ordinary.

A subset of added_token_ids: codefuse-ai/F2LLM-v2-160M flags 14 of its 26, the rest being ordinary additions the checkpoint made no promises about.

post_processor_token_ids `cached` `property` ¶

post_processor_token_ids: frozenset[int]

Return the ids of tokens the post-processor names.

E.g. {101, 102} for google-bert/bert-base-cased, which is [CLS] and [SEP], or {151645} (<|im_end|>) for codefuse-ai/F2LLM-v2-160M. You get an empty set for HuggingFaceTB/SmolLM2-135M-Instruct, which has no post-processor.

structural_ids `cached` `property` ¶

structural_ids: frozenset[int]

Return the ids that must survive for the tokenizer to keep working.

That is every added token, every token the post-processor names, plus whatever the backend declares important (typically the unk token, and the byte alphabet for byte-level tokenizers). This is what the trim keeps whether or not a corpus or a preset asks for it, and it is what the structural preset resolves to.

For codefuse-ai/F2LLM-v2-160M that is 282 ids: the 256 byte-alphabet characters plus the 26 added tokens. The three sources happily overlap, so this is their union rather than their sum. google-bert/bert-base-cased is the clearest case: it contributes [CLS]/[SEP] from the post-processor and [UNK] from the backend, and all three are already among its 5 added tokens.

structural_tokens `cached` `property` ¶

structural_tokens: set[str]

Return the token strings behind structural_ids, e.g. {"[UNK]", "[CLS]", ...} for BERT.

vocab_size `property` ¶

vocab_size: int

Return the number of distinct token ids in the tokenizer.

This is the tokenizer's own count, which is not the model's config.vocab_size: codefuse-ai/F2LLM-v2-160M reports 151,669 here while its config declares 151,936 rows. That is because the embedding matrix is padded but the vocabulary is not.

max_id `property` ¶

max_id: int

Return the largest token id in use, e.g. 151,668 for codefuse-ai/F2LLM-v2-160M.

uses_byte_level `property` ¶

uses_byte_level: bool

Return whether the tokenizer maps text through the ByteLevel alphabet.

When it does, every one of the 256 byte-alphabet characters must survive trimming or some inputs become unencodable. True for BPE checkpoints (e.g. codefuse-ai/F2LLM-v2-160M, HuggingFaceTB/SmolLM2-135M-Instruct), false for e.g. google-bert/bert-base-cased and google-t5/t5-small.

surface_forms `cached` `property` ¶

surface_forms: dict[str, str | None]

Map every vocabulary token to the text it actually stands for.

Byte-level tokens are decoded back to text, and the prefixes skeletoken reports, such as WordPiece's ## continuation marker and the ▁ character standing for a leading space, are undone. Tokens that are partial UTF-8 sequences map to None, since they stand for no well-formed text on their own.

E.g. "Ġde" -> " de" and "Ġ" -> " " for codefuse-ai/F2LLM-v2-160M, where the lone byte "¡" maps to None. For google-bert/bert-base-cased you get "##ing" -> "ing", and for google-t5/t5-small "▁the" -> " the".

chat_template_literals `property` ¶

chat_template_literals: str

Return the fixed text a chat template works with, with its Jinja removed.

The Jinja syntax is removed so what is left is every literal text the template may put around the message content, including the ones only a tool-call or system-prompt branch reaches.

Quoted strings inside the markup are kept too, since a role name often reaches the output through message['role'] == 'user', and the standard role names are added outright (see CHAT_ROLES).

Returns:

Type	Description
`str`	The literals, newline-separated, or an empty string without a template. For codefuse-ai/F2LLM-v2-160M this starts with the role names and runs on through the template's whitespace and its quoted strings. Encoding it needs 89 distinct tokens, which is what the trim has to keep.

unk_token `property` ¶

unk_token: str | None

Return the backend's "unknown" token if it declares one.

E.g. "[UNK]" for google-bert/bert-base-cased, "<unk>" for google-t5/t5-small, "<|endoftext|>" for HuggingFaceTB/SmolLM2-135M-Instruct. A byte-level BPE needs none, so codefuse-ai/F2LLM-v2-160M returns None.

from_tokenizer `classmethod` ¶

from_tokenizer(
    tokenizer: PreTrainedTokenizerFast,
    source: str | None = None,
) -> Self

Build a spec from a fast tokenizer object.

Parameters:

Name	Type	Description	Default
`tokenizer`	`PreTrainedTokenizerFast`	A `transformers` fast tokenizer.	required
`source`	`str \| None`	Optional provenance label.	`None`

Returns:

Type	Description
`Self`	The parsed spec.

Raises:

Type	Description
`ValueError`	If the object is not backed by a `tokenizers.Tokenizer`.

View source on GitHub: src/trimbed/spec.py lines 88–112

from_json_str `classmethod` ¶

from_json_str(
    payload: str, source: str | None = None
) -> Self

Build a spec from a serialised tokenizer.json.

Parameters:

Name	Type	Description	Default
`payload`	`str`	The JSON text.	required
`source`	`str \| None`	Optional provenance label.	`None`

Returns:

Type	Description
`Self`	The parsed spec.

View source on GitHub: src/trimbed/spec.py lines 114–125

from_path `classmethod` ¶

from_path(path: str | Path) -> Self

Build a spec from a tokenizer.json file on disk.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Path to the JSON file.	required

Returns:

Type	Description
`Self`	The parsed spec.

View source on GitHub: src/trimbed/spec.py lines 127–137

encode ¶

encode(text: str) -> list[int]

Return the ids this tokenizer currently produces for a piece of text.

Parameters:

Name	Type	Description	Default
`text`	`str`	The text to encode, e.g. `"De kat zat op de mat."`.	required

Returns:

Type	Description
`list[int]`	The token ids. codefuse-ai/F2LLM-v2-160M answers that example with `[1912, 44256, 1147, 266, 1179, 409, 5517, 13]`, which is `De \| Ġkat \| Ġz \| at \| Ġop \| Ġde \| Ġmat \| .`.

View source on GitHub: src/trimbed/spec.py lines 297–308

describe ¶

describe() -> dict[str, str | int | bool | None]

Return a small summary suitable for logging.

Returns:

Type	Description
`dict[str, str \| int \| bool \| None]`	A JSON-serialisable summary of the tokenizer's shape. For google-bert/bert-base-cased: `{"model_type": "WordPiece", "vocab_size": 28996, "added_tokens": 5, "special_tokens": 5, "max_token_id": 28995, "uses_byte_level": false, "unk_token": "[UNK]", "has_post_processor": true, "has_chat_template": false}`, plus `source`.

View source on GitHub: src/trimbed/spec.py lines 346–367

Tokenizer spec¶

trimbed.spec ¶

JINJA_CONSTRUCT module-attribute ¶

QUOTED_STRING module-attribute ¶

CHAT_ROLES module-attribute ¶

TokenizerSpec dataclass ¶

model property ¶

model_type property ¶

backend cached property ¶

vocabulary cached property ¶

id_to_token cached property ¶

added_tokens property ¶

added_token_ids cached property ¶

special_token_ids cached property ¶

post_processor_token_ids cached property ¶

structural_ids cached property ¶

structural_tokens cached property ¶

vocab_size property ¶

max_id property ¶

uses_byte_level property ¶

surface_forms cached property ¶

chat_template_literals property ¶

unk_token property ¶

from_tokenizer classmethod ¶

from_json_str classmethod ¶

from_path classmethod ¶

encode ¶

describe ¶

JINJA_CONSTRUCT `module-attribute` ¶

QUOTED_STRING `module-attribute` ¶

CHAT_ROLES `module-attribute` ¶

TokenizerSpec `dataclass` ¶

model `property` ¶

model_type `property` ¶

backend `cached` `property` ¶

vocabulary `cached` `property` ¶

id_to_token `cached` `property` ¶

added_tokens `property` ¶

added_token_ids `cached` `property` ¶

special_token_ids `cached` `property` ¶

post_processor_token_ids `cached` `property` ¶

structural_ids `cached` `property` ¶

structural_tokens `cached` `property` ¶

vocab_size `property` ¶

max_id `property` ¶

uses_byte_level `property` ¶

surface_forms `cached` `property` ¶

chat_template_literals `property` ¶

unk_token `property` ¶

from_tokenizer `classmethod` ¶

from_json_str `classmethod` ¶

from_path `classmethod` ¶