Backend registry¶

trimbed.backends re-exports every backend class for convenience, but each one is documented on its own page. Only the registry functions are rendered here.

trimbed.backends ¶

Registry of per-tokenizer-family trimming adapters.

Adding support for a new tokenizer family means adding one module here and decorating its class with register_backend, following one of the existing backends. Nothing else in the package needs to change.

register_backend ¶

register_backend(cls: BackendT) -> BackendT

Register a backend adapter under its declared model_type.

Parameters:

Name	Type	Description	Default
`cls`	`BackendT`	A concrete `VocabBackend` subclass.	required

Returns:

Type	Description
`BackendT`	The class unchanged, so this can be used as a decorator.

Raises:

Type	Description
`ValueError`	If the class has no `model_type` or the type is already taken.

View source on GitHub: src/trimbed/backends/__init__.py lines 16–34

get_backend ¶

get_backend(model_type: str) -> VocabBackend

Look up the adapter for a tokenizer.json model type.

Parameters:

Name	Type	Description	Default
`model_type`	`str`	Value of `model.type`, e.g. `"BPE"` for codefuse-ai/F2LLM-v2-160M or `"WordPiece"` for google-bert/bert-base-cased.	required

Returns:

Type	Description
`VocabBackend`	The registered adapter instance, e.g. a `BpeBackend`.

Raises:

Type	Description
`KeyError`	If no adapter handles that model type.

View source on GitHub: src/trimbed/backends/__init__.py lines 37–56

supported_model_types ¶

supported_model_types() -> tuple[str, ...]

Return the tokenizer model types that can currently be trimmed.

E.g. ("BPE", "Unigram", "WordLevel", "WordPiece").

View source on GitHub: src/trimbed/backends/__init__.py lines 59–64